[petsc-users] Right DM for a particle network
Matthew Knepley
knepley at gmail.com
Wed Jul 31 21:00:25 CDT 2024
On Wed, Jul 31, 2024 at 7:34 PM Marco Seiz <marco at kit.ac.jp> wrote:
> Since PETSc allows for setting of non-local matrix entries I should
> probably be able to set the "missing" entries. Would something like
>
> 1) Construct matrix A for conduction term
>
> 2) Calculate RHS as rhs = source(local information) + A * global vector
>
> 3) hand off to TS
>
> work then? Basically skipping over the DM in the first place and letting
> the matrix handle the connectivity.
>
It sounds like it could work, but now I feel I do not understand your code
enough to be certain. Particle neighborhoods will determine
row sparsity, but how will that be determined in parallel? You could move
rows to different processes with MatGetSubMatrix and then
query them, but this does not seem superior to sending particles. I think I
will not understand until I see a small prototype.
Thanks,
Matt
> Best regards,
>
> Marco
>
>
> On 31.07.24 23:28, Matthew Knepley wrote:
> > On Wed, Jul 31, 2024 at 10:08 AM Mark Adams <mfadams at lbl.gov <mailto:
> mfadams at lbl.gov>> wrote:
> >
> > Just a thought, but perhaps he may want to use just sparse matrices,
> AIJ. He Manages the connectivity And we manage ghost values.
> >
> >
> > He is reconfiguring the neighborhood (row) each time, so you would
> essentially create a new matrix at each step with different sparsity. It
> would definitely function, but I wonder if he would have enough local
> information to construct the rows?
> >
> > Thanks,
> >
> > Matt
> >
> >
> > On Wed, Jul 31, 2024 at 6:25 AM Matthew Knepley <knepley at gmail.com
> <mailto:knepley at gmail.com>> wrote:
> >
> > On Tue, Jul 30, 2024 at 11:32 PM Marco Seiz <marco at kit.ac.jp
> <mailto:marco at kit.ac.jp>> wrote:
> >
> > Hello,
> >
> > maybe to clarify a bit further: I'd essentially like to
> solve heat transport between particles only, without solving the transport
> on my voxel mesh since there's a large scale difference between the voxel
> size and the particle size and heat transport should be fast enough that
> voxel resolution is unnecessary. Basically a discrete element method just
> for heat transport. The whole motion/size change part is handled separately
> on the voxel mesh.
> > Based on the connectivity, I can make a graph (attached an
> example from a 3D case, for description see [1]) and on each vertex
> (particle) of the graph I want to account for source terms and conduction
> along the edges. What I'd like to avoid is managing the exchange for
> non-locally owned vertices during the solve (e.g. for RHS evaluation)
> myself but rather have the DM do it with DMGlobalToLocal() and friends.
> Thinking a bit further, I'd probably also want to associate some data with
> the edges since that will enter the conduction term but stays constant
> during a solve (think contact area between particles).
> >
> > Looking over the DMSwarm examples, the coupling between
> particles is via the background mesh, so e.g. I can't say "loop over all
> local particles and for each particle and its neighbours do X". I could use
> the projection part to dump the the source terms from the particles to a
> coarser background mesh but for the conduction term I don't see how I could
> get a good approximation of the contact area on the background mesh without
> having a mesh at a similar resolution as I already have, kinda destroying
> the purpose of the whole thing.
> >
> >
> > The point I was trying to make in my previous message is that
> DMSwarm does not require a background mesh. The examples use one because
> that is what we use to evaluate particle grouping. However, you have an
> independent way to do this, so you do not need it.
> >
> > Second, the issue of replicated particles. DMSwarmMigrate allows
> you to replicate particles, using the input flag. Of course, you would have
> to manage removing particles you no longer want.
> >
> > Thanks,
> >
> > Matt
> >
> >
> > [1] Points represent particles, black lines are edges, with
> the color indicating which worker "owns" the particle, with 3 workers being
> used and only a fraction of edges/vertices being displayed to keep it
> somewhat tidy. The position of the points corresponds to the particles' x-y
> position, with the z position being ignored. Particle ownership isn't done
> via looking where the particle is on the voxel grid, but rather by dividing
> the array of particle indices into subarrays, so e.g. particles [0-n/3) go
> to the first worker and so on. Since my particles can span multiple workers
> on the voxel grid this makes it much easier to update edge information with
> one-sided communication. As you can see the "mesh" is quite irregular with
> no nice boundary existing for connected particles owned by different
> workers.
> >
> > Best regards,
> > Marco
> >
> > On 30.07.24 22:56, Mark Adams wrote:
> > > * they do have a vocal mesh, so perhaps They want DM Plex.
> > >
> > > * they want ghost particle communication, that also might
> want a mesh
> > >
> > > * DM swarm does not have a notion of ghost particle, as
> far as I know, but it could use one
> > >
> > > On Tue, Jul 30, 2024 at 7:58 AM Matthew Knepley <
> knepley at gmail.com <mailto:knepley at gmail.com> <mailto:knepley at gmail.com
> <mailto:knepley at gmail.com>>> wrote:
> > >
> > > On Tue, Jul 30, 2024 at 12: 24 AM Marco Seiz <marco@
> kit. ac. jp> wrote: Hello, I'd like to solve transient heat transport at a
> particle scale using TS, with the per-particle equation being something
> like dT_i / dt = (S(T_i) + sum(F(T_j,
> > > ZjQcmQRYFpfptBannerStart
> > > __
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > >
> > > __
> > > ZjQcmQRYFpfptBannerEnd
> > > On Tue, Jul 30, 2024 at 12:24 AM Marco Seiz <
> marco at kit.ac.jp <mailto:marco at kit.ac.jp> <mailto:marco at kit.ac.jp <mailto:
> marco at kit.ac.jp>>> wrote:
> > >
> > > __
> > > Hello, I'd like to solve transient heat transport
> at a particle scale using TS, with the per-particle equation being
> something like dT_i / dt = (S(T_i) + sum(F(T_j, T_i), j connecting to i))
> with a nonlinear source term S and a conduction term
> > > ZjQcmQRYFpfptBannerStart
> > > __
> > > This Message Is From an External Sender
> > > This message came from outside your organization.
> > >
> > > __
> > > ZjQcmQRYFpfptBannerEnd
> > >
> > > Hello,
> > >
> > > I'd like to solve transient heat transport at a
> particle scale using TS, with the per-particle equation being something like
> > >
> > > dT_i / dt = (S(T_i) + sum(F(T_j, T_i), j
> connecting to i))
> > >
> > > with a nonlinear source term S and a conduction
> term F. The particles can move, deform and grow/shrink/vanish on a voxel
> grid, but for the temperature a particle-scale resolution should be
> sufficient. The particles' connectivity will change during the simulation,
> but is assumed constant during a single timestep. I have a data structure
> tracking the particles' connectivity, so I can say which particles should
> conduct heat to each other. I exploit symmetry and so only save the
> "forward" edges, so e.g. for touching particles 1->2->3, I only store [[2],
> [3], []], from which the full list [[2], [1, 3], [2]] could be
> reconstructed but which I'd like to avoid. In parallel each worker would
> own some of the particle data, so e.g. for the 1->2->3 example and 2
> workers, worker 0 could own [[2]] and worker 1 [[3],[]].
> > >
> > > Looking over the DM variants, either DMNetwork or
> some manual mesh build with DMPlex seem suited for this. I'd especially
> like it if the adjacency information is handled by the DM automagically
> based on the edges so I don't have to deal with ghost particle
> communication myself. I already tried something basic with DMNetwork,
> though for some reason the offsets I get from DMNetworkGetGlobalVecOffset()
> are larger than the actual network. I've attached what I have so far but
> comparing to e.g. src/snes/tutorials/network/ex1.c I don't see what I'm
> doing wrong if I don't need data at the edges. I might not be seeing the
> trees for the forest though. The output with -dmnetwork_view looks
> reasonable to me. Any help in fixing this approach, or if it would seem
> suitable pointers to using DMPlex for this problem, would be appreciated.
> > >
> > > To me, this sounds like you should built it with
> DMSwarm. Why?
> > >
> > > 1) We only have vertices and edges, so a mesh does not
> buy us anything.
> > >
> > > 2) You are managing the parallel particle
> connectivity, so DMPlex topology is not buying us anything. Unless I am
> misunderstanding.
> > >
> > > 3) DMNetwork has a lot of support for vertices with
> different characteristics. Your particles all have the same attributes, so
> this is unnecessary.
> > >
> > > How would you set this up?
> > >
> > > 1) Declare all particle attributes. There are many
> Swarm examples, but say ex6 which simulates particles moving under a
> central force.
> > >
> > > 2) That example decides when to move particles using a
> parallel background mesh. However, you know which particles you want to
> move,
> > > so you just change the _rank_ field to the new
> rank and call DMSwarmMigrate() with migration type _basic_.
> > >
> > > It should be straightforward to setup a tiny example
> moving around a few particles to see if it does everything you want.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > >
> > > Best regards,
> > > Marco
> > >
> > > --
> > > What most experimenters take for granted before they
> begin their experiments is infinitely more interesting than any results to
> which their experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA_i_lNEf$ <
> https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA_i_lNEf$ > <
> https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bLVHnoUGooYpdfGD8zNQrHTY2ln70W082hEc6pG7vdjA2fCvs77tcI9d7QOA0i_FjGK1of3nNOKXCE-4Rwb0$
> <
> https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bLVHnoUGooYpdfGD8zNQrHTY2ln70W082hEc6pG7vdjA2fCvs77tcI9d7QOA0i_FjGK1of3nNOKXCE-4Rwb0$
> >>
> > >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA_i_lNEf$ <
> https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA8lfrK9k$ >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA_i_lNEf$ <
> https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA8lfrK9k$ >
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA_i_lNEf$ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!YL_1KA8S6hXo_7KadaomeVGRJ3rcpPCWSKldVHeN9BX4QcQQVawkdT5T-vOLAz--lVQxj5WxRhOwA8lfrK9k$ >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240731/6b6bc08e/attachment-0001.html>
More information about the petsc-users
mailing list