[MOAB-dev] periodic boundaries
Alexander Nikolas Breuer
anbreuer at ucsd.edu
Wed Apr 20 15:05:39 CDT 2016
Dear Vijay,
thanks for checking back on me.
> If you have found a solution in the mean
> time, do let us know your approach. There is another question in the
> list that may be relevant to this discussion.
I solved the issue on my side, at least generic enough for single-node convergence benchmarks. However its definitely a “custom” solution.. Here’s what I have done:
* assign a tag to faces representing a periodic boundary
* read this in as usual: MOAB is not aware of periodic boundaries during the entire solution
* create a custom data structure containing all periodic faces
* iterate over this data structure and find the matching partner for every periodic face, this is the one sharing two vertices (in 2D). E.g. if y-dimension is constant on both sides, the x-coords have to match
* Now begins the tricky part: Getting it right for my application
* I am storing faces with ids of face-adjacent elements explicitly and independently of MOAB, here I am removing the redundant face: Don’t want to apply something twice, e.g. fluxes, to adjacent elements just because its a periodic face. However, this poses certain issues, since I am losing the associated vertices of the removed face.
* Still need this vertex info since I want to compute outer pointing-normals in a generic way. Here I am falling back to vertex coords of the elements, where I haven’t removed anything.
In short: Periodic faces are only store once, periodic vertices of elements are stored redundantly.
Alex
--
Alexander Breuer
University of California, San Diego
San Diego Supercomputer Center
9500 Gilman Drive MC 0505
La Jolla, California 92093-0505, USA
http://hpgeoc.sdsc.edu
+1 (858) 822-5692
> On Apr 12, 2016, at 10:04 AM, Vijay S. Mahadevan <vijay.m at gmail.com> wrote:
>
> Alexander, first I apologize about the long delay. This was in my
> draft and slipped my mind. If you have found a solution in the mean
> time, do let us know your approach. There is another question in the
> list that may be relevant to this discussion.
>
>>> Are you proposing to use a structured grid with triangles and tets here ?
>>
>> Would be an option to reproduce benchmarks on regular meshes with triangles and tets. Fully unstructured is better though, since I could also check that the meshing (numerics following from that) + MPI don’t hurt convergence.
>
> Ok, in that case, I don't see a value add in spending time on
> structured triangles/tets.
>
>> I guess I am almost there. Have the entity handles of the periodic faces and the entity handles of the elements on both sides. Unfortunately I am stuck at convincing MOAB that the faces now have two neighbors. Here’s what I am trying:
>
> Like I said before, the adjacency routines in MOAB are not equipped to
> handle the query at the periodic boundary. Your tag based method will
> have to use your custom tag value to relate entities on either side of
> the periodic boundary. I would think the adjacency based method should
> work out of the box and unsure why it doesn't. We may need to add few
> other things in parallel (especially in computation of the skin), but
> the serial runs should not fail.
>
> Iulian, do we have a consistency check to verify if the adjacency
> actually makes sense ? For example, if a face had 3 adjacent elements
> for whatever reason, it may be a valid interpretation in AMR.
> Similarly, for a boundary surface with only one adjacent element, if
> we add another to represent periodicity, that should work for Alex's
> case, no ?
>
> Vijay
>
> On Thu, Mar 3, 2016 at 12:28 AM, Alexander Nikolas Breuer
> <anbreuer at ucsd.edu> wrote:
>> Vijay, thanks for your input!
>>
>>> Are you proposing to use a structured grid with triangles and tets here ?
>>
>> Would be an option to reproduce benchmarks on regular meshes with triangles and tets. Fully unstructured is better though, since I could also check that the meshing (numerics following from that) + MPI don’t hurt convergence.
>>
>>> I'll have to discuss with the team to see if we can support this
>>> natively in parallel.
>>
>> Ok, great!
>>
>>> Go ahead with the Tag-based route and let me
>>> know if that suffices for your use-cases
>>
>> I guess I am almost there. Have the entity handles of the periodic faces and the entity handles of the elements on both sides. Unfortunately I am stuck at convincing MOAB that the faces now have two neighbors. Here’s what I am trying:
>>
>>> LOG(INFO) << "before";
>>> LOG(INFO) << "the two faces: " << l_faHandle << " " << l_faHandleP << " the two triangles: " << l_element[0] << " " << l_elementP[0];
>>>
>>> moab::Range l_before;
>>> l_error = m_core.get_adjacencies( &l_faHandle,
>>> 1,
>>> m_dim,
>>> true,
>>> l_before,
>>> moab::Interface::UNION );
>>> checkError(l_error);
>>>
>>> moab::Range l_beforeP;
>>> l_error = m_core.get_adjacencies( &l_faHandleP,
>>> 1,
>>> m_dim,
>>> true,
>>> l_beforeP,
>>> moab::Interface::UNION );
>>> checkError(l_error);
>>>
>>> LOG(INFO) << l_before.size() << " " << l_before[0];
>>> LOG(INFO) << l_beforeP.size() << " " << l_beforeP[0];
>>>
>>> l_error = m_core.add_adjacencies( l_faHandle, l_elementP, false );
>>> checkError( l_error );
>>>
>>> l_error = m_core.add_adjacencies( l_faHandleP, l_element, false );
>>> checkError( l_error );
>>>
>>> moab::Range l_after;
>>> l_error = m_core.get_adjacencies( &l_faHandle,
>>> 1,
>>> m_dim,
>>> true,
>>> l_after,
>>> moab::Interface::UNION );
>>> checkError(l_error);
>>>
>>> moab::Range l_afterP;
>>> l_error = m_core.get_adjacencies( &l_faHandleP,
>>> 1,
>>> m_dim,
>>> true,
>>> l_afterP,
>>> moab::Interface::UNION );
>>> checkError(l_error);
>>>
>>> LOG(INFO) << "after";
>>> LOG(INFO) << l_after.size() << " " << l_after[0];
>>> LOG(INFO) << l_afterP.size() << " " << l_afterP[0];
>>>
>>> assert(false);
>>
>> and that’s the resulting output:
>>
>>> 2016-03-02 22:10:03,226 INFO [default] initialized MOAB:
>>> 2016-03-02 22:10:03,226 INFO [default] MOAB API version 1.01
>>> 2016-03-02 22:10:03,226 INFO [default] MOAB 4.9.0
>>
>> [...]
>>> 2016-03-02 22:10:03,228 INFO [default] before
>>> 2016-03-02 22:10:03,228 INFO [default] the two faces: 1152921504606846987 1152921504606847016 the two triangles: 2305843009213693998 2305843009213694003
>>> 2016-03-02 22:10:03,228 INFO [default] 1 2305843009213693998
>>> 2016-03-02 22:10:03,228 INFO [default] 1 2305843009213694003
>>> 2016-03-02 22:10:03,228 INFO [default] after
>>> 2016-03-02 22:10:03,228 INFO [default] 1 2305843009213693998
>>> 2016-03-02 22:10:03,228 INFO [default] 1 2305843009213694003
>>
>> Am I missing something obvious here? Should I build my own lookup?
>>
>> Alex
>>
>> --
>> Alexander Breuer
>> University of California, San Diego
>> San Diego Supercomputer Center
>> 9500 Gilman Drive MC 0505
>> La Jolla, California 92093-0505, USA
>> http://hpgeoc.sdsc.edu
>> +1 (858) 822-5692
>>
>>
>>
>>> On Mar 2, 2016, at 4:25 PM, Vijay S. Mahadevan <vijay.m at gmail.com> wrote:
>>>
>>>> For the setup: I am finally aiming at Tets, but triangles are good for testing.
>>>
>>> Understood.
>>>
>>>> I also found some documentation on structured meshes using hexes in MOAB (incl. periodic boundaries), could this be extended to triangles and tets (keeping the structure alive)?
>>>
>>> Yes that's for a pure structured mesh (ScdInterface) which doesn't
>>> store any connectivity or adjacency information explicitly. You can
>>> use it for stencil computations and apply periodic B.C but it is not
>>> straightforward to extend.
>>>
>>> Are you proposing to use a structured grid with triangles and tets here ?
>>>
>>>> I am using periodic boundaries mainly for convergence benchmarks: Serial is most important, I wouldn’t say no to a MPI-version though.
>>>
>>> As I said, the unstructured mesh doesn't care what you impose as B.C.
>>> You can define sidesets as you need and create sets internally in MOAB
>>> to store the faces on the periodic boundary. Once you have that, you
>>> can internally have a map for adjacency or face-neighbors.
>>>
>>> Supporting periodicity also means that we need to do a validation when
>>> we load a mesh. If the topology doesn't match, the whole setup will
>>> fail. Of course the user needs to guarantee this if you want sensible
>>> behavior.
>>>
>>>> I need the face-neighbors of my elements: Going through surfaces is therefore the most natural choice. Could also go through vertices and find elements appearing twice (triangles) or three times (tets). Whatever is simple.
>>>
>>> I would go through surfaces. For FV/DG, that's natural.
>>>
>>>> Tags are a good idea for a workaround: I’ll tell gmsh to forward the periodic boundaries as a tag and then match via the tangential dimensions for a start.
>>>
>>> Yes. This is the simplest way to do it.
>>>
>>> I'll have to discuss with the team to see if we can support this
>>> natively in parallel. Go ahead with the Tag-based route and let me
>>> know if that suffices for your use-cases. I'll keep you updated as we
>>> continue our discussions.
>>>
>>> VIjay
>>>
>>> On Wed, Mar 2, 2016 at 1:37 PM, Alexander Nikolas Breuer
>>> <anbreuer at ucsd.edu> wrote:
>>>>> Alex, I don't think we have the necessary semantics to express
>>>>> periodic boundaries natively on an unstructured mesh in MOAB. Partly
>>>>> because, no one asked for it.
>>>>
>>>> Ok, this explains why I couldn’t find any docs on periodic boundaries ;)
>>>>
>>>> For the setup: I am finally aiming at Tets, but triangles are good for testing. It is FV/DG -> Information is shared between face-neighbors (tets) or edge-neighbors (triangles), no dependencies on the vertices. The periodic boundaries are for testing and convergence benchmarks.
>>>>
>>>> I also found some documentation on structured meshes using hexes in MOAB (incl. periodic boundaries), could this be extended to triangles and tets (keeping the structure alive)? One would “just” have to split the hexes. This way I could use structured meshes with triangles and tets for MPI, however handle them unstructured in my simulations internally. For the serial runs I would go for workaround to get true unstructured meshes in.
>>>>
>>>>> 1) Serial or parallel runs ?
>>>>> a) Serial is relatively much simpler and you can get away with MOAB
>>>>> implementation as it stands.
>>>>> b) Parallel complicates this since the elements on either side of a
>>>>> periodic boundary will be on different partitions and we need to
>>>>> modify shared entity resolution to let it know which process to send
>>>>> messages.
>>>>
>>>> I am using periodic boundaries mainly for convergence benchmarks: Serial is most important, I wouldn’t say no to a MPI-version though.
>>>>
>>>>> 2) Boundary description: nodes vs surfaces (edges)
>>>>> a) In serial, it could be as simple as matching nodes with the same
>>>>> tag value or say Y-coordinate during your setup phase and using that
>>>>> information later during computation
>>>>> b) In parallel, again, you need to rely on resolve_shared_ents to work
>>>>> correctly so that things work out of the box.
>>>>
>>>> I need the face-neighbors of my elements: Going through surfaces is therefore the most natural choice. Could also go through vertices and find elements appearing twice (triangles) or three times (tets). Whatever is simple.
>>>> Tags are a good idea for a workaround: I’ll tell gmsh to forward the periodic boundaries as a tag and then match via the tangential dimensions for a start.
>>>>
>>>>> 3) Do you need adjacency queries ?
>>>>
>>>> During runtime of my simulation? No, after the setup through MOAB all the information goes through an interface into internal structures (pointers, offsets, etc.).
>>>>
>>>>> 4) In parallel, do you expect ghost layers on the periodic boundary ?
>>>>
>>>> Yes, this is the smarter choice. Otherwise I would have to convince the partitioner not to cut along the boundaries.
>>>>
>>>> ***
>>>>
>>>> Something unrelated: Are there thread safe parts in MOAB? E.g. could I use OMP to iterate over my entities and then ask the core for information on those (read only)?
>>>>
>>>> Alex
>>>>
>>>> --
>>>> Alexander Breuer
>>>> University of California, San Diego
>>>> San Diego Supercomputer Center
>>>> 9500 Gilman Drive MC 0505
>>>> La Jolla, California 92093-0505, USA
>>>> http://hpgeoc.sdsc.edu
>>>> +1 (858) 822-5692
>>>>
>>>>
>>>>
>>>>> On Mar 2, 2016, at 9:55 AM, Vijay S. Mahadevan <vijay.m at gmail.com> wrote:
>>>>>
>>>>> Alex, I don't think we have the necessary semantics to express
>>>>> periodic boundaries natively on an unstructured mesh in MOAB. Partly
>>>>> because, no one asked for it.
>>>>>
>>>>> So couple of questions for you.
>>>>>
>>>>> 1) Serial or parallel runs ?
>>>>> a) Serial is relatively much simpler and you can get away with MOAB
>>>>> implementation as it stands.
>>>>> b) Parallel complicates this since the elements on either side of a
>>>>> periodic boundary will be on different partitions and we need to
>>>>> modify shared entity resolution to let it know which process to send
>>>>> messages.
>>>>>
>>>>> 2) Boundary description: nodes vs surfaces (edges)
>>>>> a) In serial, it could be as simple as matching nodes with the same
>>>>> tag value or say Y-coordinate during your setup phase and using that
>>>>> information later during computation
>>>>> b) In parallel, again, you need to rely on resolve_shared_ents to work
>>>>> correctly so that things work out of the box.
>>>>>
>>>>> 3) Do you need adjacency queries ?
>>>>>
>>>>> 4) In parallel, do you expect ghost layers on the periodic boundary ?
>>>>>
>>>>> Depending on what your use case is, this could be done in a variety of
>>>>> ways. Let us know.
>>>>>
>>>>> Vijay
>>>>>
>>>>> On Wed, Mar 2, 2016 at 11:29 AM, Alexander Nikolas Breuer
>>>>> <anbreuer at ucsd.edu> wrote:
>>>>>> Writing about nodes:
>>>>>>
>>>>>> I am searching the neighbors by going from triangles over edges to
>>>>>> neighboring triangles again. Let me know if it's worth trying to go through
>>>>>> the vertices for the boundary.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Mar 2, 2016, at 9:20 AM, Alexander Nikolas Breuer <anbreuer at ucsd.edu>
>>>>>> wrote:
>>>>>>
>>>>>> Uh, ok. Here's a link:
>>>>>> https://www.dropbox.com/s/f41mmkl47zxrm0a/gmsh.msh?dl=0
>>>>>>
>>>>>> The bnd condition is described in terms of nodes:
>>>>>>
>>>>>> $Periodic
>>>>>> 3
>>>>>> 0 2 1
>>>>>> 1
>>>>>> 2 1
>>>>>> 0 3 4
>>>>>> 1
>>>>>> 3 4
>>>>>> 1 2 4
>>>>>> 9
>>>>>> 14 40
>>>>>> 15 39
>>>>>> 16 38
>>>>>> 17 37
>>>>>> 18 36
>>>>>> 19 35
>>>>>> 20 34
>>>>>> 21 33
>>>>>> 22 32
>>>>>> $EndPeriodic
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Mar 2, 2016, at 6:19 AM, Grindeanu, Iulian R. <iulian at mcs.anl.gov> wrote:
>>>>>>
>>>>>> attachment was scrubbed, can you send it again ? maybe as a zip file?
>>>>>> somehow, it does not like msh suffix
>>>>>> Thanks,
>>>>>> Iulian
>>>>>> ________________________________
>>>>>> From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf
>>>>>> of Alexander Nikolas Breuer [anbreuer at ucsd.edu]
>>>>>> Sent: Tuesday, March 01, 2016 11:20 PM
>>>>>> To: moab-dev at mcs.anl.gov
>>>>>> Subject: [MOAB-dev] periodic boundaries
>>>>>>
>>>>>> Dear MOAB-team,
>>>>>>
>>>>>> I am trying to read a triangular mesh with periodic boundaries created by
>>>>>> gmsh into MOAB (attached). However I don’t see this when going through
>>>>>> MOAB’s adjacencies. Is there a way to make the mesh reader aware of periodic
>>>>>> boundaries?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alexander Breuer
>>>>>> University of California, San Diego
>>>>>> San Diego Supercomputer Center
>>>>>> 9500 Gilman Drive MC 0505
>>>>>> La Jolla, California 92093-0505, USA
>>>>>> http://hpgeoc.sdsc.edu
>>>>>> +1 (858) 822-5692
>>>>>>
>>>>>>
>>>>>>
>>>>
>>
More information about the moab-dev
mailing list