From knepley at gmail.com Mon Nov 1 04:31:49 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Nov 2021 05:31:49 -0400 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: On Sun, Oct 31, 2021 at 12:21 PM ?? wrote: > Dear Matt > > Thank you for your detailed explanation. > > First, I would like to answer your question about my application where low > dimensional features are not embedded in the larger volume. It is quite > general in structural engineering. For example, buildings are generally > modelled as shells and beams, which are two and one dimension respectively. > While building foundation is modelled by solid elements, which is three > dimension, at the same time. > I think I see what you want now. Let me make a suggestion (along the lines of what Mark said), and attempt to justify it by answering some questions. Suggestion: I think you should consider using a volumetric mesh for your problem Q1: Can I get the same algebraic system this way? Yes. You would only assign unknowns to faces and edges of the mesh where you have shell and beam elements. Q2: What are the advantages? You would get topological connectivity of all parts of the structure, automatic coupling of the different formulations, and you could separately solve the different structures using block preconditioners, while still forming a unified residual so that you can guarantee a consistent solution. Q3: Wouldn't this be expensive? No. For the shells and beams, you would still need the vertices, edges, and faces. First, by the Euler relation, these would outnumbers the additional cells. Second, since no unknowns would be associated with the cells, only additional memory in the mesh would be used, not the system solves. Memory and time are dominated by the solve, so this should be in the noise. What do you think? Thanks, Matt > Secondly, It is regrettably to find that DMComposite is not available for > me, because all my solid, shell, and beam elements are connected each other. > > At last, I have build a simple program to see if DMPlexSetCellType() works > for me, following the suggestion of functions in PETSc like > DMPlexCreateCGNS. But it failed when it tried to do DMPlexInterpolate > ! 9----------8---------13 > ! /| /| /| > ! / | / | / | > ! / | / | / | > ! 6---------7---------12 | > ! | | | | | | > ! | | | | | | > ! | | | | | | > ! | | | | | | > ! | 5------|---4-------|-11--------17--------16 > ! | / | / | / / / > ! | / | / | / / / > ! |/ |/ |/ / / > ! 2---------3---------10--------14-------15 > > The calculation result are follows. It seems that the cell type are set > correctly but depth is still 2. > -------------------------------------------------------------------- > DM Object: TestMesh 2 MPI processes > type: plex > TestMesh in 3 dimensions: > 0-cells: 16 0 > 3-cells: 20 (18) 0 > Labels: > celltype: 3 strata with value/size (7 (2), 4 (2), 0 (16)) > depth: 2 strata with value/size (0 (16), 1 (20)) > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: Array was not checked out > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.16.0-351-g743e004674 > GIT Date: 2021-10-29 09:32:23 -0500 > [0]PETSC ERROR: ./ex3f90 on a arch-linux-c-debug named DESKTOP-9ITFSBM by > hillyuan Mon Nov 1 00:26:39 2021 > [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc > --with-fc=mpiifort --with-fortran-bindings=1 > --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.3.0 > --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.3.0 > [0]PETSC ERROR: #1 DMRestoreWorkArray() at > /home/hillyuan/programs/petsc/src/dm/interface/dm.c:1580 > [0]PETSC ERROR: #2 DMPlexRestoreRawFaces_Internal() at > /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:323 > [0]PETSC ERROR: #3 DMPlexInterpolateFaces_Internal() at > /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:375 > [0]PETSC ERROR: #4 DMPlexInterpolate() at > /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:1340 > > ----------------------------------------------------------------------------------------- > I attached my test program in this mail. It is much appreciated that you > could provide any suggestion. > > Best regards, > > Yuan > > > 2021?10?31?(?) 21:16 Matthew Knepley : > >> On Thu, Oct 28, 2021 at 10:48 PM ?? wrote: >> >>> Dear Matt, >>> >>> My mesh is something like the following figure, which is composed of >>> three elements : one hexahedron(solid element), one quadrilateral (shell >>> element), and one line (beam element). I found the function "TestEmptyStrata" >>> in file \dm\impls\plex\tests\ex11.c would be a good example to read in such >>> a kind of mesh by using DMPlexSetCone. But a problem is that you should >>> declare all faces and edges of hexahedron element, all edges in >>> quadrilateral element by DMPlexSetCone, otherwise PETsc could not do >>> topological interpolation afterwards. Am I right here? >>> As general in FEM mesh, my mesh does not contain any information >>> about faces or edges of solid elements. That's why I consider using >>> DMCOMPOSITE. That is >>> >>> - Put hexahedron, quadrilateral, and line elements into different DM >>> structures. >>> - do topological interpolation in those DMs separately. >>> - composite them. >>> >>> Is there anything wrong in my above consideration? Any suggestions? >>> >>> ------------ >>> /| /| >>> / | / | cell 0: Hex >>> / | / | >>> ------------/ | >>> | | | | >>> | | | | cell 1: Quad >>> | --------|---|------------ >>> | / | / / >>> | / | / / >>> |/ |/ / >>> ------------------------------------------- >>> cell 2: line >>> >>> Much thanks for your help. >>> >> >> If you are solving something where everything is embedded in a volumetric >> mesh, then there is no problem. However, if you really have >> the mesh above, where lower dimensional pieces are sticking out of the >> mesh, then Plex can represent the mesh, but automatic interpolation >> (creation of edges and faces) will not work. Why is this? We use depth in >> the DAG as a proxy for cell dimension, but this will no longer work >> if faces are not part of a volume. >> >> Will DMCOMPOSITE do what you want? It depends. It will be able to lay out >> a vector, but it will not know about any topological connectivity >> between the meshes and will not preallocate a Jacobian with any >> interaction. If the meshes are truly separate, this is fine. If not, it is >> not that >> useful. >> >> Could you modify the existing code to support this? Yes, it would not be >> terribly difficult. When you load the mesh, you must know what kind >> of cell you are loading. You could explicitly set this using >> DMPlexSetCellType(). Then, instead of taking a certain height stratum of >> the DAG >> to loop over, you would instead use all cells marked with a certain cell >> type. The rest of the interpolation code should work fine. >> >> What kind of physics do you have where low dimensional features are not >> embedded in the larger volume? >> >> Thanks, >> >> Matt >> >> >>> Yuan >>> >>> 2021?10?28?(?) 22:05 Matthew Knepley : >>> >>>> On Thu, Oct 28, 2021 at 4:59 AM ?? wrote: >>>> >>>>> Dear Matt, >>>>> >>>>> Thank you for your quick response. >>>>> >>>>> I think what you mean is to build DAG from my mesh at first and then >>>>> call DMPlexCreateFromDAG >>>>> () >>>>> to construct DMPlex. >>>>> >>>> >>>> No, I do not mean that. >>>> >>>> >>>>> A new problem is, as I know, the function DMPlexInterpolate would >>>>> generate points with different depth. What's the difference between those >>>>> faces and segment elements generated by DMPlexInterpolate with that >>>>> defined by the original mesh, or should we not use DMPlexInterpolate in >>>>> such a case? >>>>> >>>>> On the other hand, can DMComposite be used in this case by defining >>>>> DMPlex with different topological dimensions at first and then composite >>>>> them? >>>>> >>>> >>>> You do not need that. I am obviously not understanding your question. >>>> My short answer is that Plex _already_ handles cells of different >>>> dimension automatically without anything extra. >>>> >>>> Maybe it would help if you defined a specific problem you have. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks in advance. >>>>> >>>>> Yuan >>>>> >>>>> >>>>> 2021?10?27?(?) 19:27 Matthew Knepley : >>>>> >>>>>> On Wed, Oct 27, 2021 at 4:50 AM ?? wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am trying to parallelize my serial FEM program using PETSc. This >>>>>>> program calculates structure deformation by using various types of elements >>>>>>> such as solid, shell, beam, and truss. At the very beginning, I found it >>>>>>> was hard for me to put such kinds of elements into DMPlex. Because solid >>>>>>> elements are topologically three dimensional, shell element two, and beam >>>>>>> or truss are topologically one-dimensional elements. After reading chapter >>>>>>> 2.10: "DMPlex: Unstructured Grids in PETSc" of users manual carefully, I >>>>>>> found the provided functions, such as DMPlexSetCone, cannot declare those >>>>>>> topological differences. >>>>>>> >>>>>>> My question is : Is it possible and how to define all those >>>>>>> topologically different elements into a DMPlex struct? >>>>>>> >>>>>> >>>>>> Yes. The idea is to program in a dimension-independent way, so that >>>>>> the code can handle cells of any dimension. >>>>>> What you probably want is the "depth" in the DAG representation, >>>>>> which you can think of as the dimension of a cell. >>>>>> >>>>>> >>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetPointDepth.html#DMPlexGetPointDepth >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks in advance! >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Yuan. >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Mon Nov 1 09:21:41 2021 From: hongzhang at anl.gov (Zhang, Hong) Date: Mon, 1 Nov 2021 14:21:41 +0000 Subject: [petsc-users] Strange behavior of TS after setting hand-coded Jacobian In-Reply-To: References: <0C6ACBF3-F457-4BFD-AD19-8C455444748F@petsc.dev> <9CC15214-4601-4554-808F-C3E96DC3D34A@petsc.dev> Message-ID: <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> You can avoid the zero-diagonal elements in your case. Just remove the boundary points and the \hat{\Lambda}_r from the solution vector Y (eq(17) in your PDE file). You can calculate them separately in your RHS function (when computing the residual vector for 2 <=i<=N-1). It is not necessary to form a DAE. You can actually form an ODE instead. Hong (Mr.) > On Oct 31, 2021, at 5:38 AM, ?? wrote: > > Thanks for your suggestions and experience sharing. > > But it seems that you have missed my point. This kind of zero-diagonal > element that I described earlier is somehow inevitable. > If interested in this problem, you may refer to the PDF file that was > attached in the previous post. > > Regards > > Yu > > Zhang, Hong ?2021?10?29??? ??10:05??? >> >> One way to avoid the zero element in Jacobian is to exclude the boundary point from the solution vector. I often do this for Dirichlet boundary conditions since the value at the boundary is given directly and does not need to be taken as a degree of freedom. >> >> Hong (Mr.) >> >> On Oct 28, 2021, at 9:49 PM, ?? wrote: >> >> Thanks for your careful inspection and thoughtful suggestions. >> >>> finite differencing may put a small non-zero value in that location due to numerical round-off >> >> I think your explanation is reasonable. This numerical round-off may somehow help to avoid this pivot issue. >> >> The structure of my jacobian matrix looks like this (generated by '-mat_view draw'): >> >> Analytically, the first diagonal element of the jacobian is indeed 0, as its corresponding residual function is solely determined from boundary condition of another variable. This seems a little bit wired but is mathematically well-posed. For more description about the background physics, please refer to attached PDF file for more detailed explanation on the discretization and boundary conditions. >> >> Actually, the jacobian matrix is not singular, but I do believe this numerical difficulty is caused by the zero-element in diagonal. >> In this regard, I've performed some trial and test. It seems that several methods have been worked out for this pivot issue: >> a) By setting '-pc_type svd', PETSC does not panic any more with my hand-coded jacobian, and converged solution is obtained. Efficiency is also preserved. >> b) By setting '-pc_type none', converged solution is also obtained, but it takes too many KSP iterations to converge per SNES step (usually hundreds), making the overall solution procedure very slow. >> >> Do you think these methods really solved this kind of pivot issue? Not by chance like the numerical round-off in finite difference previously. >> >> Regards >> >> Yu Cang >> >> Barry Smith ?2021?10?27??? ??9:43??? >>> >>> >>> You can run with -ksp_error_if_not_converged to get it to stop as soon as a linear solve fails to help track down the exact breaking point. >>> >>>> The problem under consideration contains an eigen-value to be solved, >>>> making the first diagonal element of the jacobian matrix being zero. >>>> From these outputs, it seems that the PC failed to factorize, which is >>>> caused by this 0 diagonal element. But I'm wondering why it works >>>> with jacobian matrix generated by finite-difference? >>> >>> Presumably your "exact" Jacobian puts a zero on the diagonal while the finite differencing may put a small non-zero value in that location due to numerical round-off. In that case even if the factorization succeeds it may produce an inaccurate solution if the value on the diagonal is very small. >>> >>> If your matrix is singular or cannot be factored with LU then you need to use a different solver for the linear system that will be robust to the zero on the diagonal. What is the structure of your Jacobian? (The analytic form). >>> >>> Barry >>> >>> >>>> On Oct 27, 2021, at 1:47 AM, ?? wrote: >>>> >>>> Thanks for your kind reply. >>>> >>>> Several comparison tests have been performed. Attached are execution >>>> output files. Below are corresponding descriptions. >>>> >>>> good.txt -- Run without hand-coded jacobian, solution converged, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason'; >>>> jac1.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian'; >>>> jac2.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >>>> -ksp_view'; >>>> jac3.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >>>> -ksp_view -ts_max_snes_failures -1 '; >>>> >>>> The problem under consideration contains an eigen-value to be solved, >>>> making the first diagonal element of the jacobian matrix being zero. >>>> From these outputs, it seems that the PC failed to factorize, which is >>>> caused by this 0 diagonal element. But I'm wondering why it works >>>> with jacobian matrix generated by finite-difference? Would employing >>>> DMDA for discretization be helpful? >>>> >>>> Regards >>>> >>>> Yu Cang >>>> >>>> Barry Smith ?2021?10?25??? ??10:50??? >>>>> >>>>> >>>>> It is definitely unexpected that -snes_test_jacobian verifies the Jacobian as matching but the solve process is completely different. >>>>> >>>>> Please run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian and send all the output >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Oct 25, 2021, at 9:53 AM, ?? wrote: >>>>>> >>>>>> I'm using TS to solve a set of DAE, which originates from a >>>>>> one-dimensional problem. The grid points are uniformly distributed. >>>>>> For simplicity, the DMDA is not employed for discretization. >>>>>> >>>>>> At first, only the residual function is prescribed through >>>>>> 'TSSetIFunction', and PETSC produces converged results. However, after >>>>>> providing hand-coded Jacobian through 'TSSetIJacobian', the internal >>>>>> SNES object fails (residual norm does not change), and TS reports >>>>>> 'DIVERGED_STEP_REJECTED'. >>>>>> >>>>>> I have tried to add the option '-snes_test_jacobian' to see if the >>>>>> hand-coded jacobian is somewhere wrong, but it shows '||J - >>>>>> Jfd||_F/||J||_F = 1.07488e-10, ||J - Jfd||_F = 2.14458e-07', >>>>>> indicating that the hand-coded jacobian is correct. >>>>>> >>>>>> Then, I added a monitor for the internal SNES object through >>>>>> 'SNESMonitorSet', in which the solution vector will be displayed at >>>>>> each iteration. It is interesting to find that, if the jacobian is not >>>>>> provided, meaning finite-difference is utilized for jacobian >>>>>> evaluation internally, the solution vector converges to steady >>>>>> solution and the SNES residual norm is reduced continuously. However, >>>>>> it turns out that, as long as the jacobian is provided, the solution >>>>>> vector will NEVER get changed! So the solution procedure stucked! >>>>>> >>>>>> This is quite strange! Hope to get some advice. >>>>>> PETSC version=3.14.6, program run in serial mode. >>>>>> >>>>>> Regards >>>>>> >>>>>> Yu Cang >>>>> >>>> >>> >> >> >> From knepley at gmail.com Mon Nov 1 14:25:43 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 1 Nov 2021 15:25:43 -0400 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <631832bb-4953-a3eb-13c6-0f7fe17e869d@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> Message-ID: On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland < Eric.Chamberland at giref.ulaval.ca> wrote: > Hi Matthew, > > we do not know if DMPlexNaturalToGlobalBegin/End is buggy or if it is our > comprehension of what it should do that is wrong... > > Would you just check if what we try to do from line 313 to 356 is good or > wrong? > > The expected result is that the global vector "lGlobalVec" contains the > element numbers from the initial partition that have been put into > "lNatVec". > > Thanks a lot for any insights!! > > Okay, I have found the problem.I confess to not actually looking very closely at the implementation before. GlobalToNatural was assuming that the "natural" was also sequential, and looking at it all the tests are eq-to-parallel distribution. I need to fix it for a parallel natural state. Hopefully it will not take me long. Thanks, Matt > Eric > > > On 2021-10-27 2:32 p.m., Eric Chamberland wrote: > > Hi Matthew, > > we continued the example. Now it must be our misuse of PETSc that > produced the wrong result. > > As stated into the code: > > // The call to DMPlexNaturalToGlobalBegin/End does not produce our > expected result... > // In lGlobalVec, we expect to have: > /* > * Process [0] > * 2. > * 4. > * 8. > * 3. > * 9. > * Process [1] > * 1. > * 5. > * 7. > * 0. > * 6. > * > * but we obtained: > * > * Process [0] > * 2. > * 4. > * 8. > * 0. > * 0. > * Process [1] > * 0. > * 0. > * 0. > * 0. > * 0. > */ > > (see attached ex44.c) > > Thanks, > > Eric > On 2021-10-27 1:25 p.m., Eric Chamberland wrote: > > Great! > > Thanks Matthew, it is working for me up to that point! > > We are continuing the ex44.c and forward it to you at the next blocking > point... > > Eric > On 2021-10-27 11:14 a.m., Matthew Knepley wrote: > > On Wed, Oct 27, 2021 at 8:29 AM Eric Chamberland < > Eric.Chamberland at giref.ulaval.ca> wrote: > >> Hi Matthew, >> >> the smallest mesh which crashes the code is a 2x5 mesh: >> >> See the modified ex44.c >> >> With smaller meshes(2x2, 2x4, etc), it passes... But it bugs latter when >> I try to use DMPlexNaturalToGlobalBegin but let's keep that other problem >> for later... >> >> Thanks a lot for helping digging into this! :) >> > I have made a small fix in this branch > > https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n > > It seems to run for me. Can you check it? > > Thanks, > > Matt > > >> Eric >> >> (sorry if you received this for a 2nd times, I have trouble with my mail) >> On 2021-10-26 4:35 p.m., Matthew Knepley wrote: >> >> On Tue, Oct 26, 2021 at 1:35 PM Eric Chamberland < >> Eric.Chamberland at giref.ulaval.ca> wrote: >> >>> Here is a screenshot of the partition I hard coded (top) and >>> vertices/element numbers (down): >>> >>> I have not yet modified the ex44.c example to properly assign the >>> coordinates... >>> >>> (but I would not have done it like it is in the last version because the >>> sCoords array is the global array with global vertices number) >>> >>> I will have time to do this tomorrow... >>> >>> Maybe I can first try to reproduce all this with a smaller mesh? >>> >> >> That might make it easier to find a problem. >> >> Thanks! >> >> Matt >> >> >>> Eric >>> On 2021-10-26 9:46 a.m., Matthew Knepley wrote: >>> >>> Okay, I ran it. Something seems off with the mesh. First, I cannot >>> simply explain the partition. The number of shared vertices and edges >>> does not seem to come from a straight cut. Second, the mesh look >>> scrambled on output. >>> >>> Thanks, >>> >>> Matt >>> >>> On Sun, Oct 24, 2021 at 11:49 PM Eric Chamberland < >>> Eric.Chamberland at giref.ulaval.ca> wrote: >>> >>>> Hi Matthew, >>>> >>>> ok, I started back from your ex44.c example and added the global array >>>> of coordinates. I just have to code the creation of the local coordinates >>>> now. >>>> >>>> Eric >>>> On 2021-10-20 6:55 p.m., Matthew Knepley wrote: >>>> >>>> On Wed, Oct 20, 2021 at 3:06 PM Eric Chamberland < >>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>> >>>>> Hi Matthew, >>>>> >>>>> we tried to reproduce the error in a simple example. >>>>> >>>>> The context is the following: We hard coded the mesh and initial >>>>> partition into the code (see sConnectivity and sInitialPartition) for 2 >>>>> ranks and try to create a section in order to use the >>>>> DMPlexNaturalToGlobalBegin function to retreive our initial element numbers. >>>>> >>>>> Now the call to DMPlexDistribute give different errors depending on >>>>> what type of component we ask the field to be created. For our objective, >>>>> we would like a global field to be created on elements only (like a P0 >>>>> interpolation). >>>>> >>>>> We now have the following error generated: >>>>> >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>> [0]PETSC ERROR: Inconsistency in indices, 18 should be 17 >>>>> [0]PETSC ERROR: See >>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>> shooting. >>>>> [0]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021 >>>>> [0]PETSC ERROR: ./bug on a named rohan by ericc Wed Oct 20 14:52:36 >>>>> 2021 >>>>> [0]PETSC ERROR: Configure options >>>>> --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 --with-mpi-compilers=1 >>>>> --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 --with-cxx-dialect=C++14 >>>>> --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes >>>>> --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 >>>>> --download-ml=yes --download-mumps=yes --download-superlu=yes >>>>> --download-hpddm=yes --download-slepc=yes --download-superlu_dist=yes >>>>> --download-parmetis=yes --download-ptscotch=yes --download-metis=yes >>>>> --download-strumpack=yes --download-suitesparse=yes --download-hypre=yes >>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>> --with-scalapack=1 >>>>> --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include >>>>> --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>> -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" >>>>> [0]PETSC ERROR: #1 PetscSFCreateSectionSF() at >>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 >>>>> [0]PETSC ERROR: #2 DMPlexCreateGlobalToNaturalSF() at >>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 >>>>> [0]PETSC ERROR: #3 DMPlexDistribute() at >>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 >>>>> [0]PETSC ERROR: #4 main() at bug_section.cc:159 >>>>> [0]PETSC ERROR: No PETSc Option Table entries >>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>> entire error message to petsc-maint at mcs.anl.gov---------- >>>>> >>>>> Hope the attached code is self-explaining, note that to make it short, >>>>> we have not included the final part of it, just the buggy part we are >>>>> encountering right now... >>>>> >>>>> Thanks for your insights, >>>>> >>>> Thanks for making the example. I tweaked it slightly. I put in a test >>>> case that just makes a parallel 7 x 10 quad mesh. This works >>>> fine. Thus I think it must be something connected with the original >>>> mesh. It is hard to get a handle on it without the coordinates. >>>> Do you think you could put the coordinate array in? I have added the >>>> code to load them (see attached file). >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>>> Eric >>>>> On 2021-10-06 9:23 p.m., Matthew Knepley wrote: >>>>> >>>>> On Wed, Oct 6, 2021 at 5:43 PM Eric Chamberland < >>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>> >>>>>> Hi Matthew, >>>>>> >>>>>> we tried to use that. Now, we discovered that: >>>>>> >>>>>> 1- even if we "ask" for sfNatural creation with DMSetUseNatural, it >>>>>> is not created because DMPlexCreateGlobalToNaturalSF looks for a "section": >>>>>> this is not documented in DMSetUseNaturalso we are asking ourselfs: "is >>>>>> this a permanent feature or a temporary situation?" >>>>>> >>>>> I think explaining this will help clear up a lot. >>>>> >>>>> What the Natural2Global map does is permute a solution vector into the >>>>> ordering that it would have had prior to mesh distribution. >>>>> Now, in order to do this permutation, I need to know the original >>>>> (global) data layout. If it is not specified _before_ distribution, we >>>>> cannot build the permutation. The section describes the data layout, >>>>> so I need it before distribution. >>>>> >>>>> I cannot think of another way that you would implement this, but if >>>>> you want something else, let me know. >>>>> >>>>>> 2- We then tried to create a "section" in different manners: we took >>>>>> the code into the example petsc/src/dm/impls/plex/tests/ex15.c. However, >>>>>> we ended up with a segfault: >>>>>> >>>>>> corrupted size vs. prev_size >>>>>> [rohan:07297] *** Process received signal *** >>>>>> [rohan:07297] Signal: Aborted (6) >>>>>> [rohan:07297] Signal code: (-6) >>>>>> [rohan:07297] [ 0] /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >>>>>> [rohan:07297] [ 1] /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >>>>>> [rohan:07297] [ 2] /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >>>>>> [rohan:07297] [ 3] /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >>>>>> [rohan:07297] [ 4] /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >>>>>> [rohan:07297] [ 5] /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >>>>>> [rohan:07297] [ 6] /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >>>>>> [rohan:07297] [ 7] /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >>>>>> [rohan:07297] [ 8] >>>>>> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >>>>>> [rohan:07297] [ 9] >>>>>> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >>>>>> [rohan:07297] [10] >>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >>>>>> [rohan:07297] [11] >>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >>>>>> [rohan:07297] [12] >>>>>> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >>>>>> [rohan:07297] [13] /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >>>>>> >>>>>> [rohan:07297] [14] >>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >>>>>> [rohan:07297] [15] >>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >>>>>> [rohan:07297] [16] >>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >>>>>> [rohan:07297] [17] >>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >>>>>> [rohan:07297] [18] >>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >>>>>> >>>>> I am not sure what happened here, but if you could send a sample code, >>>>> I will figure it out. >>>>> >>>>>> If we do not create a section, the call to DMPlexDistribute is >>>>>> successful, but DMPlexGetGlobalToNaturalSF return a null SF pointer... >>>>>> >>>>> Yes, it just ignores it in this case because it does not have a global >>>>> layout. >>>>> >>>>>> Here are the operations we are calling ( this is almost the code we >>>>>> are using, I just removed verifications and creation of the connectivity >>>>>> which use our parallel structure and code): >>>>>> >>>>>> =========== >>>>>> >>>>>> PetscInt* lCells = 0; >>>>>> PetscInt lNumCorners = 0; >>>>>> PetscInt lDimMail = 0; >>>>>> PetscInt lnumCells = 0; >>>>>> >>>>>> //At this point we create the cells for PETSc expected input for >>>>>> DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and lnumCells >>>>>> to correct values. >>>>>> ... >>>>>> >>>>>> DM lDMBete = 0 >>>>>> DMPlexCreate(lMPIComm,&lDMBete); >>>>>> >>>>>> DMSetDimension(lDMBete, lDimMail); >>>>>> >>>>>> DMPlexBuildFromCellListParallel(lDMBete, >>>>>> lnumCells, >>>>>> PETSC_DECIDE, >>>>>> >>>>>> pLectureElementsLocaux.reqNbTotalSommets(), >>>>>> lNumCorners, >>>>>> lCells, >>>>>> PETSC_NULL); >>>>>> >>>>>> DM lDMBeteInterp = 0; >>>>>> DMPlexInterpolate(lDMBete, &lDMBeteInterp); >>>>>> DMDestroy(&lDMBete); >>>>>> lDMBete = lDMBeteInterp; >>>>>> >>>>>> DMSetUseNatural(lDMBete,PETSC_TRUE); >>>>>> >>>>>> PetscSF lSFMigrationSansOvl = 0; >>>>>> PetscSF lSFMigrationOvl = 0; >>>>>> DM lDMDistribueSansOvl = 0; >>>>>> DM lDMAvecOverlap = 0; >>>>>> >>>>>> PetscPartitioner lPart; >>>>>> DMPlexGetPartitioner(lDMBete, &lPart); >>>>>> PetscPartitionerSetFromOptions(lPart); >>>>>> >>>>>> PetscSection section; >>>>>> PetscInt numFields = 1; >>>>>> PetscInt numBC = 0; >>>>>> PetscInt numComp[1] = {1}; >>>>>> PetscInt numDof[4] = {1, 0, 0, 0}; >>>>>> PetscInt bcFields[1] = {0}; >>>>>> IS bcPoints[1] = {NULL}; >>>>>> >>>>>> DMSetNumFields(lDMBete, numFields); >>>>>> >>>>>> DMPlexCreateSection(lDMBete, NULL, numComp, numDof, numBC, >>>>>> bcFields, bcPoints, NULL, NULL, §ion); >>>>>> DMSetLocalSection(lDMBete, section); >>>>>> >>>>>> DMPlexDistribute(lDMBete, 0, &lSFMigrationSansOvl, >>>>>> &lDMDistribueSansOvl); // segfault! >>>>>> >>>>>> =========== >>>>>> >>>>>> So we have other question/remarks: >>>>>> >>>>>> 3- Maybe PETSc expect something specific that is missing/not >>>>>> verified: for example, we didn't gave any coordinates since we just want to >>>>>> partition and compute overlap for the mesh... and then recover our element >>>>>> numbers in a "simple way" >>>>>> >>>>>> 4- We are telling ourselves it is somewhat a "big price to pay" to >>>>>> have to build an unused section to have the global to natural ordering set >>>>>> ? Could this requirement be avoided? >>>>>> >>>>> I don't think so. There would have to be _some_ way of describing your >>>>> data layout in terms of mesh points, and I do not see how you could use >>>>> less memory doing that. >>>>> >>>>>> 5- Are there any improvement towards our usages in 3.16 release? >>>>>> >>>>> Let me try and run the code above. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>>> Thanks, >>>>>> >>>>>> Eric >>>>>> >>>>>> >>>>>> On 2021-09-29 7:39 p.m., Matthew Knepley wrote: >>>>>> >>>>>> On Wed, Sep 29, 2021 at 5:18 PM Eric Chamberland < >>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I come back with _almost_ the original question: >>>>>>> >>>>>>> I would like to add an integer information (*our* original element >>>>>>> number, not petsc one) on each element of the DMPlex I create with >>>>>>> DMPlexBuildFromCellListParallel. >>>>>>> >>>>>>> I would like this interger to be distribruted by or the same way >>>>>>> DMPlexDistribute distribute the mesh. >>>>>>> >>>>>>> Is it possible to do this? >>>>>>> >>>>>> >>>>>> I think we already have support for what you want. If you call >>>>>> >>>>>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >>>>>> >>>>>> before DMPlexDistribute(), it will compute a PetscSF encoding the >>>>>> global to natural map. You >>>>>> can get it with >>>>>> >>>>>> >>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >>>>>> >>>>>> and use it with >>>>>> >>>>>> >>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >>>>>> >>>>>> Is this sufficient? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> On 2021-07-14 1:18 p.m., Eric Chamberland wrote: >>>>>>> > Hi, >>>>>>> > >>>>>>> > I want to use DMPlexDistribute from PETSc for computing >>>>>>> overlapping >>>>>>> > and play with the different partitioners supported. >>>>>>> > >>>>>>> > However, after calling DMPlexDistribute, I noticed the elements >>>>>>> are >>>>>>> > renumbered and then the original number is lost. >>>>>>> > >>>>>>> > What would be the best way to keep track of the element >>>>>>> renumbering? >>>>>>> > >>>>>>> > a) Adding an optional parameter to let the user retrieve a vector >>>>>>> or >>>>>>> > "IS" giving the old number? >>>>>>> > >>>>>>> > b) Adding a DMLabel (seems a wrong good solution) >>>>>>> > >>>>>>> > c) Other idea? >>>>>>> > >>>>>>> > Of course, I don't want to loose performances with the need of >>>>>>> this >>>>>>> > "mapping"... >>>>>>> > >>>>>>> > Thanks, >>>>>>> > >>>>>>> > Eric >>>>>>> > >>>>>>> -- >>>>>>> Eric Chamberland, ing., M. Ing >>>>>>> Professionnel de recherche >>>>>>> GIREF/Universit? Laval >>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Universit? Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Universit? Laval >>>>> (418) 656-2131 poste 41 22 42 >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbnbhlbilhmjdpfg.png Type: image/png Size: 42972 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eejjfmbjimlkboec.png Type: image/png Size: 87901 bytes Desc: not available URL: From Eric.Chamberland at giref.ulaval.ca Mon Nov 1 16:24:29 2021 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Mon, 1 Nov 2021 17:24:29 -0400 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <631832bb-4953-a3eb-13c6-0f7fe17e869d@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> Message-ID: <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> Hi Matthew, ok, good news then! :) Meanwhile, I tested it the "other way" with DMPlexGlobalToNaturalBegin/End and it does not give a "sensed" result neither. Probably it does something like "GlobalToSequential" too? Thanks! Eric On 2021-11-01 3:25 p.m., Matthew Knepley wrote: > On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland > > wrote: > > Hi Matthew, > > we do not know if DMPlexNaturalToGlobalBegin/End is buggy or if it > is our comprehension of what it should do that is wrong... > > Would you just check if what we try to do from line 313 to 356 is > good or wrong? > > The expected result is that the global vector "lGlobalVec" > contains the element numbers from the initial partition that have > been put into "lNatVec". > > Thanks a lot for any insights!! > > Okay, I have found the problem.I confess to not actually looking very > closely at the implementation before. GlobalToNatural was assuming that > the "natural" was also sequential, and looking at it all the tests are > eq-to-parallel distribution. I need to fix it for a parallel natural > state. Hopefully > it will not take me long. > > ? Thanks, > > ? ? ?Matt > > Eric > > > On 2021-10-27 2:32 p.m., Eric Chamberland wrote: >> >> Hi Matthew, >> >> we continued the example.? Now it must be our misuse of PETSc >> that produced the wrong result. >> >> As stated into the code: >> >> // The call to DMPlexNaturalToGlobalBegin/End does not produce >> our expected result... >> ? // In lGlobalVec, we expect to have: >> ? /* >> ?? * Process [0] >> ?? * 2. >> ?? * 4. >> ?? * 8. >> ?? * 3. >> ?? * 9. >> ?? * Process [1] >> ?? * 1. >> ?? * 5. >> ?? * 7. >> ?? * 0. >> ?? * 6. >> ?? * >> ?? * but we obtained: >> ?? * >> ?? * Process [0] >> ?? * 2. >> ?? * 4. >> ?? * 8. >> ?? * 0. >> ?? * 0. >> ?? * Process [1] >> ?? * 0. >> ?? * 0. >> ?? * 0. >> ?? * 0. >> ?? * 0. >> ?? */ >> >> (see attached ex44.c) >> >> Thanks, >> >> Eric >> >> On 2021-10-27 1:25 p.m., Eric Chamberland wrote: >>> >>> Great! >>> >>> Thanks Matthew, it is working for me up to that point! >>> >>> We are continuing the ex44.c and forward it to you at the next >>> blocking point... >>> >>> Eric >>> >>> On 2021-10-27 11:14 a.m., Matthew Knepley wrote: >>>> On Wed, Oct 27, 2021 at 8:29 AM Eric Chamberland >>>> >>> > wrote: >>>> >>>> Hi Matthew, >>>> >>>> the smallest mesh which crashes the code is a 2x5 mesh: >>>> >>>> See the modified ex44.c >>>> >>>> With smaller meshes(2x2, 2x4, etc), it passes...? But it >>>> bugs latter when I try to use DMPlexNaturalToGlobalBegin >>>> but let's keep that other problem for later... >>>> >>>> Thanks a lot for helping digging into this! :) >>>> >>>> I have made a small fix in this branch >>>> >>>> https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n >>>> >>>> >>>> It seems to run for me. Can you check it? >>>> >>>> ? Thanks, >>>> >>>> ? ? ?Matt >>>> >>>> Eric >>>> >>>> (sorry if you received this for a? 2nd times, I have >>>> trouble with my mail) >>>> >>>> On 2021-10-26 4:35 p.m., Matthew Knepley wrote: >>>>> On Tue, Oct 26, 2021 at 1:35 PM Eric Chamberland >>>>> >>>> > wrote: >>>>> >>>>> Here is a screenshot of the partition I hard coded >>>>> (top) and vertices/element numbers (down): >>>>> >>>>> I have not yet modified the ex44.c example to properly >>>>> assign the coordinates... >>>>> >>>>> (but I would not have done it like it is in the last >>>>> version because the sCoords array is the global array >>>>> with global vertices number) >>>>> >>>>> I will have time to do this tomorrow... >>>>> >>>>> Maybe I can first try to reproduce all this with a >>>>> smaller mesh? >>>>> >>>>> >>>>> That might make it easier to find a problem. >>>>> >>>>> ? Thanks! >>>>> >>>>> ? ? ?Matt >>>>> >>>>> Eric >>>>> >>>>> On 2021-10-26 9:46 a.m., Matthew Knepley wrote: >>>>>> Okay, I ran it. Something seems off with the mesh. >>>>>> First, I cannot simply explain the partition. The >>>>>> number of shared vertices and edges >>>>>> does not seem to come from a straight cut. Second, >>>>>> the mesh look scrambled on output. >>>>>> >>>>>> ? Thanks, >>>>>> >>>>>> ? ? Matt >>>>>> >>>>>> On Sun, Oct 24, 2021 at 11:49 PM Eric Chamberland >>>>>> >>>>> > wrote: >>>>>> >>>>>> Hi Matthew, >>>>>> >>>>>> ok, I started back from your ex44.c example and >>>>>> added the global array of coordinates.? I just >>>>>> have to code the creation of the local >>>>>> coordinates now. >>>>>> >>>>>> Eric >>>>>> >>>>>> On 2021-10-20 6:55 p.m., Matthew Knepley wrote: >>>>>>> On Wed, Oct 20, 2021 at 3:06 PM Eric Chamberland >>>>>>> >>>>>> > wrote: >>>>>>> >>>>>>> Hi Matthew, >>>>>>> >>>>>>> we tried to reproduce the error in a simple >>>>>>> example. >>>>>>> >>>>>>> The context is the following: We hard coded >>>>>>> the mesh and initial partition into the code >>>>>>> (see sConnectivity and sInitialPartition) >>>>>>> for 2 ranks and try to create a section in >>>>>>> order to use the DMPlexNaturalToGlobalBegin >>>>>>> function to retreive our initial element >>>>>>> numbers. >>>>>>> >>>>>>> Now the call to DMPlexDistribute give >>>>>>> different errors depending on what type of >>>>>>> component we ask the field to be created.? >>>>>>> For our objective, we would like a global >>>>>>> field to be created on elements only (like a >>>>>>> P0 interpolation). >>>>>>> >>>>>>> We now have the following error generated: >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error >>>>>>> Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Petsc has generated >>>>>>> inconsistent data >>>>>>> [0]PETSC ERROR: Inconsistency in indices, 18 >>>>>>> should be 17 >>>>>>> [0]PETSC ERROR: See >>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>> >>>>>>> for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version >>>>>>> 3.15.0, Mar 30, 2021 >>>>>>> [0]PETSC ERROR: ./bug on a? named rohan by >>>>>>> ericc Wed Oct 20 14:52:36 2021 >>>>>>> [0]PETSC ERROR: Configure options >>>>>>> --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 >>>>>>> --with-mpi-compilers=1 >>>>>>> --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 >>>>>>> --with-cxx-dialect=C++14 --with-make-np=12 >>>>>>> --with-shared-libraries=1 >>>>>>> --with-debugging=yes --with-memalign=64 >>>>>>> --with-visibility=0 --with-64-bit-indices=0 >>>>>>> --download-ml=yes --download-mumps=yes >>>>>>> --download-superlu=yes --download-hpddm=yes >>>>>>> --download-slepc=yes >>>>>>> --download-superlu_dist=yes >>>>>>> --download-parmetis=yes >>>>>>> --download-ptscotch=yes --download-metis=yes >>>>>>> --download-strumpack=yes >>>>>>> --download-suitesparse=yes >>>>>>> --download-hypre=yes >>>>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>> --with-scalapack=1 >>>>>>> --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include >>>>>>> --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>> -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" >>>>>>> [0]PETSC ERROR: #1 PetscSFCreateSectionSF() >>>>>>> at >>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 >>>>>>> [0]PETSC ERROR: #2 >>>>>>> DMPlexCreateGlobalToNaturalSF() at >>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 >>>>>>> [0]PETSC ERROR: #3 DMPlexDistribute() at >>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 >>>>>>> [0]PETSC ERROR: #4 main() at bug_section.cc:159 >>>>>>> [0]PETSC ERROR: No PETSc Option Table entries >>>>>>> [0]PETSC ERROR: ----------------End of Error >>>>>>> Message -------send entire error message to >>>>>>> petsc-maint at mcs.anl.gov >>>>>>> ---------- >>>>>>> >>>>>>> Hope the attached code is self-explaining, >>>>>>> note that to make it short, we have not >>>>>>> included the final part of it, just the >>>>>>> buggy part we are encountering right now... >>>>>>> >>>>>>> Thanks for your insights, >>>>>>> >>>>>>> Thanks for making the example. I tweaked it >>>>>>> slightly. I put in a test case that just makes a >>>>>>> parallel 7 x 10 quad mesh. This works >>>>>>> fine. Thus I think it must be something >>>>>>> connected with the original mesh. It is hard to >>>>>>> get a handle on it without the coordinates. >>>>>>> Do you think you could put the coordinate array >>>>>>> in? I have added the code to load them (see >>>>>>> attached file). >>>>>>> >>>>>>> ? Thanks, >>>>>>> >>>>>>> ? ? ?Matt >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> On 2021-10-06 9:23 p.m., Matthew Knepley wrote: >>>>>>>> On Wed, Oct 6, 2021 at 5:43 PM Eric >>>>>>>> Chamberland >>>>>>>> >>>>>>> > >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Matthew, >>>>>>>> >>>>>>>> we tried to use that. Now, we >>>>>>>> discovered that: >>>>>>>> >>>>>>>> 1- even if we "ask" for sfNatural >>>>>>>> creation with DMSetUseNatural, it is >>>>>>>> not created because >>>>>>>> DMPlexCreateGlobalToNaturalSF looks for >>>>>>>> a "section": this is not documented in >>>>>>>> DMSetUseNaturalso we are asking >>>>>>>> ourselfs: "is this a permanent feature >>>>>>>> or a temporary situation?" >>>>>>>> >>>>>>>> I think explaining this will help clear up >>>>>>>> a lot. >>>>>>>> >>>>>>>> What the Natural2Global?map does is permute >>>>>>>> a solution vector into the ordering that it >>>>>>>> would have had prior to mesh distribution. >>>>>>>> Now, in order to do this permutation, I >>>>>>>> need to know the original (global) data >>>>>>>> layout. If it is not specified _before_ >>>>>>>> distribution, we >>>>>>>> cannot build the permutation. The section >>>>>>>> describes the data layout, so I need it >>>>>>>> before distribution. >>>>>>>> >>>>>>>> I cannot think of another way that you >>>>>>>> would implement this, but if you want >>>>>>>> something else, let me know. >>>>>>>> >>>>>>>> 2- We then tried to create a "section" >>>>>>>> in different manners: we took the code >>>>>>>> into the example >>>>>>>> petsc/src/dm/impls/plex/tests/ex15.c. >>>>>>>> However, we ended up with a segfault: >>>>>>>> >>>>>>>> corrupted size vs. prev_size >>>>>>>> [rohan:07297] *** Process received >>>>>>>> signal *** >>>>>>>> [rohan:07297] Signal: Aborted (6) >>>>>>>> [rohan:07297] Signal code: (-6) >>>>>>>> [rohan:07297] [ 0] >>>>>>>> /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >>>>>>>> [rohan:07297] [ 1] >>>>>>>> /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >>>>>>>> [rohan:07297] [ 2] >>>>>>>> /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >>>>>>>> [rohan:07297] [ 3] >>>>>>>> /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >>>>>>>> [rohan:07297] [ 4] >>>>>>>> /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >>>>>>>> [rohan:07297] [ 5] >>>>>>>> /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >>>>>>>> [rohan:07297] [ 6] >>>>>>>> /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >>>>>>>> [rohan:07297] [ 7] >>>>>>>> /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >>>>>>>> [rohan:07297] [ 8] >>>>>>>> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >>>>>>>> [rohan:07297] [ 9] >>>>>>>> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >>>>>>>> [rohan:07297] [10] >>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >>>>>>>> [rohan:07297] [11] >>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >>>>>>>> [rohan:07297] [12] >>>>>>>> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >>>>>>>> [rohan:07297] [13] >>>>>>>> /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >>>>>>>> >>>>>>>> [rohan:07297] [14] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >>>>>>>> [rohan:07297] [15] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >>>>>>>> [rohan:07297] [16] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >>>>>>>> [rohan:07297] [17] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >>>>>>>> [rohan:07297] [18] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >>>>>>>> >>>>>>>> I am not sure what happened here, but if >>>>>>>> you could send a sample code, I will figure >>>>>>>> it out. >>>>>>>> >>>>>>>> If we do not create a section, the call >>>>>>>> to DMPlexDistribute is successful, but >>>>>>>> DMPlexGetGlobalToNaturalSF return a >>>>>>>> null SF pointer... >>>>>>>> >>>>>>>> Yes, it just ignores it in this case >>>>>>>> because it does not have a global layout. >>>>>>>> >>>>>>>> Here are the operations we are calling >>>>>>>> ( this is almost the code we are using, >>>>>>>> I just removed verifications and >>>>>>>> creation of the connectivity which use >>>>>>>> our parallel structure and code): >>>>>>>> >>>>>>>> =========== >>>>>>>> >>>>>>>> ? PetscInt* lCells????? = 0; >>>>>>>> ? PetscInt lNumCorners = 0; >>>>>>>> ? PetscInt lDimMail??? = 0; >>>>>>>> ? PetscInt lnumCells?? = 0; >>>>>>>> >>>>>>>> ? //At this point we create the cells >>>>>>>> for PETSc expected input for >>>>>>>> DMPlexBuildFromCellListParallel and set >>>>>>>> lNumCorners, lDimMail and lnumCells to >>>>>>>> correct values. >>>>>>>> ? ... >>>>>>>> >>>>>>>> ? DM lDMBete = 0 >>>>>>>> DMPlexCreate(lMPIComm,&lDMBete); >>>>>>>> >>>>>>>> DMSetDimension(lDMBete, lDimMail); >>>>>>>> >>>>>>>> DMPlexBuildFromCellListParallel(lDMBete, >>>>>>>> ????????????????????????????????? >>>>>>>> lnumCells, >>>>>>>> ????????????????????????????????? >>>>>>>> PETSC_DECIDE, >>>>>>>> pLectureElementsLocaux.reqNbTotalSommets(), >>>>>>>> ????????????????????????????????? >>>>>>>> lNumCorners, >>>>>>>> ????????????????????????????????? lCells, >>>>>>>> ????????????????????????????????? >>>>>>>> PETSC_NULL); >>>>>>>> >>>>>>>> ? DM lDMBeteInterp = 0; >>>>>>>> DMPlexInterpolate(lDMBete, &lDMBeteInterp); >>>>>>>> DMDestroy(&lDMBete); >>>>>>>> ? lDMBete = lDMBeteInterp; >>>>>>>> >>>>>>>> DMSetUseNatural(lDMBete,PETSC_TRUE); >>>>>>>> >>>>>>>> ? PetscSF lSFMigrationSansOvl = 0; >>>>>>>> ? PetscSF lSFMigrationOvl = 0; >>>>>>>> ? DM lDMDistribueSansOvl = 0; >>>>>>>> ? DM lDMAvecOverlap = 0; >>>>>>>> >>>>>>>> PetscPartitioner lPart; >>>>>>>> DMPlexGetPartitioner(lDMBete, &lPart); >>>>>>>> PetscPartitionerSetFromOptions(lPart); >>>>>>>> >>>>>>>> PetscSection section; >>>>>>>> PetscInt numFields?? = 1; >>>>>>>> PetscInt numBC?????? = 0; >>>>>>>> PetscInt numComp[1]? = {1}; >>>>>>>> PetscInt numDof[4]?? = {1, 0, 0, 0}; >>>>>>>> PetscInt bcFields[1] = {0}; >>>>>>>> IS bcPoints[1] = {NULL}; >>>>>>>> >>>>>>>> DMSetNumFields(lDMBete, numFields); >>>>>>>> >>>>>>>> DMPlexCreateSection(lDMBete, NULL, >>>>>>>> numComp, numDof, numBC, bcFields, >>>>>>>> bcPoints, NULL, NULL, §ion); >>>>>>>> DMSetLocalSection(lDMBete, section); >>>>>>>> >>>>>>>> DMPlexDistribute(lDMBete, 0, >>>>>>>> &lSFMigrationSansOvl, >>>>>>>> &lDMDistribueSansOvl); // segfault! >>>>>>>> >>>>>>>> =========== >>>>>>>> >>>>>>>> So we have other question/remarks: >>>>>>>> >>>>>>>> 3- Maybe PETSc expect something >>>>>>>> specific that is missing/not verified: >>>>>>>> for example, we didn't gave any >>>>>>>> coordinates since we just want to >>>>>>>> partition and compute overlap for the >>>>>>>> mesh... and then recover our element >>>>>>>> numbers in a "simple way" >>>>>>>> >>>>>>>> 4- We are telling ourselves it is >>>>>>>> somewhat a "big price to pay" to have >>>>>>>> to build an unused section to have the >>>>>>>> global to natural ordering set ?? Could >>>>>>>> this requirement be avoided? >>>>>>>> >>>>>>>> I don't think so. There would have to be >>>>>>>> _some_ way of describing your data layout >>>>>>>> in terms of mesh points, and I do not see >>>>>>>> how you could use less memory doing that. >>>>>>>> >>>>>>>> 5- Are there any improvement towards >>>>>>>> our usages in 3.16 release? >>>>>>>> >>>>>>>> Let me try and run the code above. >>>>>>>> >>>>>>>> ? Thanks, >>>>>>>> >>>>>>>> ?Matt >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> >>>>>>>> On 2021-09-29 7:39 p.m., Matthew >>>>>>>> Knepley wrote: >>>>>>>>> On Wed, Sep 29, 2021 at 5:18 PM Eric >>>>>>>>> Chamberland >>>>>>>>> >>>>>>>> > >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I come back with _almost_ the >>>>>>>>> original question: >>>>>>>>> >>>>>>>>> I would like to add an integer >>>>>>>>> information (*our* original element >>>>>>>>> number, not petsc one) on each >>>>>>>>> element of the DMPlex I create with >>>>>>>>> DMPlexBuildFromCellListParallel. >>>>>>>>> >>>>>>>>> I would like this interger to be >>>>>>>>> distribruted by or the same way >>>>>>>>> DMPlexDistribute distribute the mesh. >>>>>>>>> >>>>>>>>> Is it possible to do this? >>>>>>>>> >>>>>>>>> >>>>>>>>> I think we already have support for >>>>>>>>> what you want. If you call >>>>>>>>> >>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >>>>>>>>> >>>>>>>>> >>>>>>>>> before DMPlexDistribute(), it will >>>>>>>>> compute a PetscSF encoding the global >>>>>>>>> to natural map. You >>>>>>>>> can get it with >>>>>>>>> >>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >>>>>>>>> >>>>>>>>> >>>>>>>>> and use it with >>>>>>>>> >>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >>>>>>>>> >>>>>>>>> >>>>>>>>> Is this sufficient? >>>>>>>>> >>>>>>>>> ? Thanks, >>>>>>>>> >>>>>>>>> ? ? ?Matt >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> On 2021-07-14 1:18 p.m., Eric >>>>>>>>> Chamberland wrote: >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I want to use DMPlexDistribute >>>>>>>>> from PETSc for computing overlapping >>>>>>>>> > and play with the different >>>>>>>>> partitioners supported. >>>>>>>>> > >>>>>>>>> > However, after calling >>>>>>>>> DMPlexDistribute, I noticed the >>>>>>>>> elements are >>>>>>>>> > renumbered and then the original >>>>>>>>> number is lost. >>>>>>>>> > >>>>>>>>> > What would be the best way to >>>>>>>>> keep track of the element renumbering? >>>>>>>>> > >>>>>>>>> > a) Adding an optional parameter >>>>>>>>> to let the user retrieve a vector or >>>>>>>>> > "IS" giving the old number? >>>>>>>>> > >>>>>>>>> > b) Adding a DMLabel (seems a >>>>>>>>> wrong good solution) >>>>>>>>> > >>>>>>>>> > c) Other idea? >>>>>>>>> > >>>>>>>>> > Of course, I don't want to loose >>>>>>>>> performances with the need of this >>>>>>>>> > "mapping"... >>>>>>>>> > >>>>>>>>> > Thanks, >>>>>>>>> > >>>>>>>>> > Eric >>>>>>>>> > >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Universit? Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for >>>>>>>>> granted before they begin their >>>>>>>>> experiments is infinitely more >>>>>>>>> interesting than any results to which >>>>>>>>> their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Universit? Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted >>>>>>>> before they begin their experiments is >>>>>>>> infinitely more interesting than any >>>>>>>> results to which their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Eric Chamberland, ing., M. Ing >>>>>>> Professionnel de recherche >>>>>>> GIREF/Universit? Laval >>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before >>>>>>> they begin their experiments is infinitely more >>>>>>> interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Universit? Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they >>>>>> begin their experiments is infinitely more >>>>>> interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Universit? Laval >>>>> (418) 656-2131 poste 41 22 42 >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin >>>>> their experiments is infinitely more interesting than any >>>>> results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin >>>> their experiments is infinitely more interesting than any >>>> results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Universit? Laval (418) 656-2131 poste 41 22 42 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbnbhlbilhmjdpfg.png Type: image/png Size: 42972 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eejjfmbjimlkboec.png Type: image/png Size: 87901 bytes Desc: not available URL: From yuanxi at advancesoft.jp Mon Nov 1 22:20:04 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Tue, 2 Nov 2021 12:20:04 +0900 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: Dear Matthew, I built a test problem using the strategy you suggested. It works! It is enough for me right now. Thank you very much. ! 9----------8----------13 ! /| /| /| ! / | / | / | ! / | / | / | ! 6---------7---------12 | ! | | | | | | ! | | | | | | ! | | | | | | ! | | | | | | ! | 5------|---4-------|--11--------17------16 ! | / | / | / / / ! | / | / | / / / ! |/ |/ |/ / / ! 2-- ------3---------10--------14-------15 coneSize = (/ 8,8,8,8, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 /) cones = (/ 2,5,4,3,6,7,8,9, 3,4,11,10,7,12,13,8, & 10,14,17,11, 10,14,17,11, 14,15,16,17, 14,15,16,17 /) There is still a problem left, however, that is how to build a 3D cell from a 2 vertex segment. I think I could construct a virtual tetrahedron. Finally, It can't be denied that this approach needs additional effort in mesh construction (and It is something strange to construct a 3D object from a one dimensional segment). In fact, structures with different topological dimensions are not that rare. Parachute, for example, may be composed with two dimensional parafoil and one dimensional cord. FRP(Fiber reinforced plastics) may be modelled by one dimensional reinforcement and three dimensional plastics. It is therefore, I am wondering, if PETSc would take into consider of such cases by, for example - Enable a face, DM_POLYTOPE_SEGMENT2D, defined by one edge. And - Eable cells, such as DM_POLYTOPE_QUADRILATERAL3D, DM_POLYTOPE_TRIANGLE3D and DM_POLYTOPE_SEGMENT3D, with one face. Sorry for a newbie in PETSc to provide such a suggestion. But if it could be accomplished, it would help mech structural engineering, civil engineering, and solid mechanics applicationers. Best regards, Yuan 2021?11?1?(?) 18:32 Matthew Knepley : > On Sun, Oct 31, 2021 at 12:21 PM ?? wrote: > >> Dear Matt >> >> Thank you for your detailed explanation. >> >> First, I would like to answer your question about my application where >> low dimensional features are not embedded in the larger volume. It is quite >> general in structural engineering. For example, buildings are generally >> modelled as shells and beams, which are two and one dimension respectively. >> While building foundation is modelled by solid elements, which is three >> dimension, at the same time. >> > > I think I see what you want now. Let me make a suggestion (along the lines > of what Mark said), and attempt to justify it by answering some questions. > > Suggestion: I think you should consider using a volumetric mesh for your > problem > > Q1: Can I get the same algebraic system this way? > > Yes. You would only assign unknowns to faces and edges of the mesh where > you have shell and beam elements. > > Q2: What are the advantages? > > You would get topological connectivity of all parts of the structure, > automatic coupling of the different formulations, > and you could separately solve the different structures using block > preconditioners, while still forming a unified > residual so that you can guarantee a consistent solution. > > Q3: Wouldn't this be expensive? > > No. For the shells and beams, you would still need the vertices, edges, > and faces. First, by the Euler relation, these would > outnumbers the additional cells. Second, since no unknowns would be > associated with the cells, only additional memory in > the mesh would be used, not the system solves. Memory and time are > dominated by the solve, so this should be in the noise. > > What do you think? > > Thanks, > > Matt > > >> Secondly, It is regrettably to find that DMComposite is not available for >> me, because all my solid, shell, and beam elements are connected each other. >> >> At last, I have build a simple program to see if DMPlexSetCellType() >> works for me, following the suggestion of functions in PETSc like >> DMPlexCreateCGNS. But it failed when it tried to do DMPlexInterpolate >> ! 9----------8---------13 >> ! /| /| /| >> ! / | / | / | >> ! / | / | / | >> ! 6---------7---------12 | >> ! | | | | | | >> ! | | | | | | >> ! | | | | | | >> ! | | | | | | >> ! | 5------|---4-------|-11--------17--------16 >> ! | / | / | / / / >> ! | / | / | / / / >> ! |/ |/ |/ / / >> ! 2---------3---------10--------14-------15 >> >> The calculation result are follows. It seems that the cell type are set >> correctly but depth is still 2. >> -------------------------------------------------------------------- >> DM Object: TestMesh 2 MPI processes >> type: plex >> TestMesh in 3 dimensions: >> 0-cells: 16 0 >> 3-cells: 20 (18) 0 >> Labels: >> celltype: 3 strata with value/size (7 (2), 4 (2), 0 (16)) >> depth: 2 strata with value/size (0 (16), 1 (20)) >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: Array was not checked out >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.16.0-351-g743e004674 >> GIT Date: 2021-10-29 09:32:23 -0500 >> [0]PETSC ERROR: ./ex3f90 on a arch-linux-c-debug named DESKTOP-9ITFSBM by >> hillyuan Mon Nov 1 00:26:39 2021 >> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc >> --with-fc=mpiifort --with-fortran-bindings=1 >> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.3.0 >> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.3.0 >> [0]PETSC ERROR: #1 DMRestoreWorkArray() at >> /home/hillyuan/programs/petsc/src/dm/interface/dm.c:1580 >> [0]PETSC ERROR: #2 DMPlexRestoreRawFaces_Internal() at >> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:323 >> [0]PETSC ERROR: #3 DMPlexInterpolateFaces_Internal() at >> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:375 >> [0]PETSC ERROR: #4 DMPlexInterpolate() at >> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:1340 >> >> ----------------------------------------------------------------------------------------- >> I attached my test program in this mail. It is much appreciated that you >> could provide any suggestion. >> >> Best regards, >> >> Yuan >> >> >> 2021?10?31?(?) 21:16 Matthew Knepley : >> >>> On Thu, Oct 28, 2021 at 10:48 PM ?? wrote: >>> >>>> Dear Matt, >>>> >>>> My mesh is something like the following figure, which is composed >>>> of three elements : one hexahedron(solid element), one quadrilateral (shell >>>> element), and one line (beam element). I found the function "TestEmptyStrata" >>>> in file \dm\impls\plex\tests\ex11.c would be a good example to read in such >>>> a kind of mesh by using DMPlexSetCone. But a problem is that you should >>>> declare all faces and edges of hexahedron element, all edges in >>>> quadrilateral element by DMPlexSetCone, otherwise PETsc could not do >>>> topological interpolation afterwards. Am I right here? >>>> As general in FEM mesh, my mesh does not contain any information >>>> about faces or edges of solid elements. That's why I consider using >>>> DMCOMPOSITE. That is >>>> >>>> - Put hexahedron, quadrilateral, and line elements into different >>>> DM structures. >>>> - do topological interpolation in those DMs separately. >>>> - composite them. >>>> >>>> Is there anything wrong in my above consideration? Any suggestions? >>>> >>>> ------------ >>>> /| /| >>>> / | / | cell 0: Hex >>>> / | / | >>>> ------------/ | >>>> | | | | >>>> | | | | cell 1: Quad >>>> | --------|---|------------ >>>> | / | / / >>>> | / | / / >>>> |/ |/ / >>>> ------------------------------------------- >>>> cell 2: line >>>> >>>> Much thanks for your help. >>>> >>> >>> If you are solving something where everything is embedded in a >>> volumetric mesh, then there is no problem. However, if you really have >>> the mesh above, where lower dimensional pieces are sticking out of the >>> mesh, then Plex can represent the mesh, but automatic interpolation >>> (creation of edges and faces) will not work. Why is this? We use depth >>> in the DAG as a proxy for cell dimension, but this will no longer work >>> if faces are not part of a volume. >>> >>> Will DMCOMPOSITE do what you want? It depends. It will be able to lay >>> out a vector, but it will not know about any topological connectivity >>> between the meshes and will not preallocate a Jacobian with any >>> interaction. If the meshes are truly separate, this is fine. If not, it is >>> not that >>> useful. >>> >>> Could you modify the existing code to support this? Yes, it would not be >>> terribly difficult. When you load the mesh, you must know what kind >>> of cell you are loading. You could explicitly set this using >>> DMPlexSetCellType(). Then, instead of taking a certain height stratum of >>> the DAG >>> to loop over, you would instead use all cells marked with a certain cell >>> type. The rest of the interpolation code should work fine. >>> >>> What kind of physics do you have where low dimensional features are not >>> embedded in the larger volume? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Yuan >>>> >>>> 2021?10?28?(?) 22:05 Matthew Knepley : >>>> >>>>> On Thu, Oct 28, 2021 at 4:59 AM ?? wrote: >>>>> >>>>>> Dear Matt, >>>>>> >>>>>> Thank you for your quick response. >>>>>> >>>>>> I think what you mean is to build DAG from my mesh at first and then >>>>>> call DMPlexCreateFromDAG >>>>>> () >>>>>> to construct DMPlex. >>>>>> >>>>> >>>>> No, I do not mean that. >>>>> >>>>> >>>>>> A new problem is, as I know, the function DMPlexInterpolate would >>>>>> generate points with different depth. What's the difference between those >>>>>> faces and segment elements generated by DMPlexInterpolate with that >>>>>> defined by the original mesh, or should we not use DMPlexInterpolate in >>>>>> such a case? >>>>>> >>>>>> On the other hand, can DMComposite be used in this case by defining >>>>>> DMPlex with different topological dimensions at first and then composite >>>>>> them? >>>>>> >>>>> >>>>> You do not need that. I am obviously not understanding your question. >>>>> My short answer is that Plex _already_ handles cells of different >>>>> dimension automatically without anything extra. >>>>> >>>>> Maybe it would help if you defined a specific problem you have. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks in advance. >>>>>> >>>>>> Yuan >>>>>> >>>>>> >>>>>> 2021?10?27?(?) 19:27 Matthew Knepley : >>>>>> >>>>>>> On Wed, Oct 27, 2021 at 4:50 AM ?? wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I am trying to parallelize my serial FEM program using PETSc. This >>>>>>>> program calculates structure deformation by using various types of elements >>>>>>>> such as solid, shell, beam, and truss. At the very beginning, I found it >>>>>>>> was hard for me to put such kinds of elements into DMPlex. Because solid >>>>>>>> elements are topologically three dimensional, shell element two, and beam >>>>>>>> or truss are topologically one-dimensional elements. After reading chapter >>>>>>>> 2.10: "DMPlex: Unstructured Grids in PETSc" of users manual carefully, I >>>>>>>> found the provided functions, such as DMPlexSetCone, cannot declare those >>>>>>>> topological differences. >>>>>>>> >>>>>>>> My question is : Is it possible and how to define all those >>>>>>>> topologically different elements into a DMPlex struct? >>>>>>>> >>>>>>> >>>>>>> Yes. The idea is to program in a dimension-independent way, so that >>>>>>> the code can handle cells of any dimension. >>>>>>> What you probably want is the "depth" in the DAG representation, >>>>>>> which you can think of as the dimension of a cell. >>>>>>> >>>>>>> >>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetPointDepth.html#DMPlexGetPointDepth >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks in advance! >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Yuan. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 2 04:16:30 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Nov 2021 05:16:30 -0400 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <631832bb-4953-a3eb-13c6-0f7fe17e869d@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> Message-ID: On Mon, Nov 1, 2021 at 5:24 PM Eric Chamberland < Eric.Chamberland at giref.ulaval.ca> wrote: > Hi Matthew, > > ok, good news then! :) > > Meanwhile, I tested it the "other way" with DMPlexGlobalToNaturalBegin/End > and it does not give a "sensed" result neither. > > Probably it does something like "GlobalToSequential" too? > I am not sure how it worked before, but I think I have it fixed. I get the output you expect for the test. I force-pushed the branch so you will have to check it out again. Thanks! Matt > Thanks! > > Eric > > > On 2021-11-01 3:25 p.m., Matthew Knepley wrote: > > On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland < > Eric.Chamberland at giref.ulaval.ca> wrote: > >> Hi Matthew, >> >> we do not know if DMPlexNaturalToGlobalBegin/End is buggy or if it is our >> comprehension of what it should do that is wrong... >> >> Would you just check if what we try to do from line 313 to 356 is good or >> wrong? >> >> The expected result is that the global vector "lGlobalVec" contains the >> element numbers from the initial partition that have been put into >> "lNatVec". >> >> Thanks a lot for any insights!! >> > Okay, I have found the problem.I confess to not actually looking very > closely at the implementation before. GlobalToNatural was assuming that > the "natural" was also sequential, and looking at it all the tests are > eq-to-parallel distribution. I need to fix it for a parallel natural state. > Hopefully > it will not take me long. > > Thanks, > > Matt > >> Eric >> >> >> On 2021-10-27 2:32 p.m., Eric Chamberland wrote: >> >> Hi Matthew, >> >> we continued the example. Now it must be our misuse of PETSc that >> produced the wrong result. >> >> As stated into the code: >> >> // The call to DMPlexNaturalToGlobalBegin/End does not produce our >> expected result... >> // In lGlobalVec, we expect to have: >> /* >> * Process [0] >> * 2. >> * 4. >> * 8. >> * 3. >> * 9. >> * Process [1] >> * 1. >> * 5. >> * 7. >> * 0. >> * 6. >> * >> * but we obtained: >> * >> * Process [0] >> * 2. >> * 4. >> * 8. >> * 0. >> * 0. >> * Process [1] >> * 0. >> * 0. >> * 0. >> * 0. >> * 0. >> */ >> >> (see attached ex44.c) >> >> Thanks, >> >> Eric >> On 2021-10-27 1:25 p.m., Eric Chamberland wrote: >> >> Great! >> >> Thanks Matthew, it is working for me up to that point! >> >> We are continuing the ex44.c and forward it to you at the next blocking >> point... >> >> Eric >> On 2021-10-27 11:14 a.m., Matthew Knepley wrote: >> >> On Wed, Oct 27, 2021 at 8:29 AM Eric Chamberland < >> Eric.Chamberland at giref.ulaval.ca> wrote: >> >>> Hi Matthew, >>> >>> the smallest mesh which crashes the code is a 2x5 mesh: >>> >>> See the modified ex44.c >>> >>> With smaller meshes(2x2, 2x4, etc), it passes... But it bugs latter >>> when I try to use DMPlexNaturalToGlobalBegin but let's keep that other >>> problem for later... >>> >>> Thanks a lot for helping digging into this! :) >>> >> I have made a small fix in this branch >> >> https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n >> >> It seems to run for me. Can you check it? >> >> Thanks, >> >> Matt >> >> >>> Eric >>> >>> (sorry if you received this for a 2nd times, I have trouble with my >>> mail) >>> On 2021-10-26 4:35 p.m., Matthew Knepley wrote: >>> >>> On Tue, Oct 26, 2021 at 1:35 PM Eric Chamberland < >>> Eric.Chamberland at giref.ulaval.ca> wrote: >>> >>>> Here is a screenshot of the partition I hard coded (top) and >>>> vertices/element numbers (down): >>>> >>>> I have not yet modified the ex44.c example to properly assign the >>>> coordinates... >>>> >>>> (but I would not have done it like it is in the last version because >>>> the sCoords array is the global array with global vertices number) >>>> >>>> I will have time to do this tomorrow... >>>> >>>> Maybe I can first try to reproduce all this with a smaller mesh? >>>> >>> >>> That might make it easier to find a problem. >>> >>> Thanks! >>> >>> Matt >>> >>> >>>> Eric >>>> On 2021-10-26 9:46 a.m., Matthew Knepley wrote: >>>> >>>> Okay, I ran it. Something seems off with the mesh. First, I cannot >>>> simply explain the partition. The number of shared vertices and edges >>>> does not seem to come from a straight cut. Second, the mesh look >>>> scrambled on output. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> On Sun, Oct 24, 2021 at 11:49 PM Eric Chamberland < >>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>> >>>>> Hi Matthew, >>>>> >>>>> ok, I started back from your ex44.c example and added the global array >>>>> of coordinates. I just have to code the creation of the local coordinates >>>>> now. >>>>> >>>>> Eric >>>>> On 2021-10-20 6:55 p.m., Matthew Knepley wrote: >>>>> >>>>> On Wed, Oct 20, 2021 at 3:06 PM Eric Chamberland < >>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>> >>>>>> Hi Matthew, >>>>>> >>>>>> we tried to reproduce the error in a simple example. >>>>>> >>>>>> The context is the following: We hard coded the mesh and initial >>>>>> partition into the code (see sConnectivity and sInitialPartition) for 2 >>>>>> ranks and try to create a section in order to use the >>>>>> DMPlexNaturalToGlobalBegin function to retreive our initial element numbers. >>>>>> >>>>>> Now the call to DMPlexDistribute give different errors depending on >>>>>> what type of component we ask the field to be created. For our objective, >>>>>> we would like a global field to be created on elements only (like a P0 >>>>>> interpolation). >>>>>> >>>>>> We now have the following error generated: >>>>>> >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>> [0]PETSC ERROR: Inconsistency in indices, 18 should be 17 >>>>>> [0]PETSC ERROR: See >>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>> shooting. >>>>>> [0]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021 >>>>>> [0]PETSC ERROR: ./bug on a named rohan by ericc Wed Oct 20 14:52:36 >>>>>> 2021 >>>>>> [0]PETSC ERROR: Configure options >>>>>> --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 --with-mpi-compilers=1 >>>>>> --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 --with-cxx-dialect=C++14 >>>>>> --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes >>>>>> --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 >>>>>> --download-ml=yes --download-mumps=yes --download-superlu=yes >>>>>> --download-hpddm=yes --download-slepc=yes --download-superlu_dist=yes >>>>>> --download-parmetis=yes --download-ptscotch=yes --download-metis=yes >>>>>> --download-strumpack=yes --download-suitesparse=yes --download-hypre=yes >>>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>> --with-scalapack=1 >>>>>> --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include >>>>>> --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>> -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" >>>>>> [0]PETSC ERROR: #1 PetscSFCreateSectionSF() at >>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 >>>>>> [0]PETSC ERROR: #2 DMPlexCreateGlobalToNaturalSF() at >>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 >>>>>> [0]PETSC ERROR: #3 DMPlexDistribute() at >>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 >>>>>> [0]PETSC ERROR: #4 main() at bug_section.cc:159 >>>>>> [0]PETSC ERROR: No PETSc Option Table entries >>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>>> entire error message to petsc-maint at mcs.anl.gov---------- >>>>>> >>>>>> Hope the attached code is self-explaining, note that to make it >>>>>> short, we have not included the final part of it, just the buggy part we >>>>>> are encountering right now... >>>>>> >>>>>> Thanks for your insights, >>>>>> >>>>> Thanks for making the example. I tweaked it slightly. I put in a test >>>>> case that just makes a parallel 7 x 10 quad mesh. This works >>>>> fine. Thus I think it must be something connected with the original >>>>> mesh. It is hard to get a handle on it without the coordinates. >>>>> Do you think you could put the coordinate array in? I have added the >>>>> code to load them (see attached file). >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>>> Eric >>>>>> On 2021-10-06 9:23 p.m., Matthew Knepley wrote: >>>>>> >>>>>> On Wed, Oct 6, 2021 at 5:43 PM Eric Chamberland < >>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>> >>>>>>> Hi Matthew, >>>>>>> >>>>>>> we tried to use that. Now, we discovered that: >>>>>>> >>>>>>> 1- even if we "ask" for sfNatural creation with DMSetUseNatural, it >>>>>>> is not created because DMPlexCreateGlobalToNaturalSF looks for a "section": >>>>>>> this is not documented in DMSetUseNaturalso we are asking ourselfs: "is >>>>>>> this a permanent feature or a temporary situation?" >>>>>>> >>>>>> I think explaining this will help clear up a lot. >>>>>> >>>>>> What the Natural2Global map does is permute a solution vector into >>>>>> the ordering that it would have had prior to mesh distribution. >>>>>> Now, in order to do this permutation, I need to know the original >>>>>> (global) data layout. If it is not specified _before_ distribution, we >>>>>> cannot build the permutation. The section describes the data layout, >>>>>> so I need it before distribution. >>>>>> >>>>>> I cannot think of another way that you would implement this, but if >>>>>> you want something else, let me know. >>>>>> >>>>>>> 2- We then tried to create a "section" in different manners: we took >>>>>>> the code into the example petsc/src/dm/impls/plex/tests/ex15.c. However, >>>>>>> we ended up with a segfault: >>>>>>> >>>>>>> corrupted size vs. prev_size >>>>>>> [rohan:07297] *** Process received signal *** >>>>>>> [rohan:07297] Signal: Aborted (6) >>>>>>> [rohan:07297] Signal code: (-6) >>>>>>> [rohan:07297] [ 0] /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >>>>>>> [rohan:07297] [ 1] /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >>>>>>> [rohan:07297] [ 2] /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >>>>>>> [rohan:07297] [ 3] /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >>>>>>> [rohan:07297] [ 4] /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >>>>>>> [rohan:07297] [ 5] /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >>>>>>> [rohan:07297] [ 6] /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >>>>>>> [rohan:07297] [ 7] /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >>>>>>> [rohan:07297] [ 8] >>>>>>> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >>>>>>> [rohan:07297] [ 9] >>>>>>> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >>>>>>> [rohan:07297] [10] >>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >>>>>>> [rohan:07297] [11] >>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >>>>>>> [rohan:07297] [12] >>>>>>> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >>>>>>> [rohan:07297] [13] /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >>>>>>> >>>>>>> [rohan:07297] [14] >>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >>>>>>> [rohan:07297] [15] >>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >>>>>>> [rohan:07297] [16] >>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >>>>>>> [rohan:07297] [17] >>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >>>>>>> [rohan:07297] [18] >>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >>>>>>> >>>>>> I am not sure what happened here, but if you could send a sample >>>>>> code, I will figure it out. >>>>>> >>>>>>> If we do not create a section, the call to DMPlexDistribute is >>>>>>> successful, but DMPlexGetGlobalToNaturalSF return a null SF pointer... >>>>>>> >>>>>> Yes, it just ignores it in this case because it does not have a >>>>>> global layout. >>>>>> >>>>>>> Here are the operations we are calling ( this is almost the code we >>>>>>> are using, I just removed verifications and creation of the connectivity >>>>>>> which use our parallel structure and code): >>>>>>> >>>>>>> =========== >>>>>>> >>>>>>> PetscInt* lCells = 0; >>>>>>> PetscInt lNumCorners = 0; >>>>>>> PetscInt lDimMail = 0; >>>>>>> PetscInt lnumCells = 0; >>>>>>> >>>>>>> //At this point we create the cells for PETSc expected input for >>>>>>> DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and lnumCells >>>>>>> to correct values. >>>>>>> ... >>>>>>> >>>>>>> DM lDMBete = 0 >>>>>>> DMPlexCreate(lMPIComm,&lDMBete); >>>>>>> >>>>>>> DMSetDimension(lDMBete, lDimMail); >>>>>>> >>>>>>> DMPlexBuildFromCellListParallel(lDMBete, >>>>>>> lnumCells, >>>>>>> PETSC_DECIDE, >>>>>>> >>>>>>> pLectureElementsLocaux.reqNbTotalSommets(), >>>>>>> lNumCorners, >>>>>>> lCells, >>>>>>> PETSC_NULL); >>>>>>> >>>>>>> DM lDMBeteInterp = 0; >>>>>>> DMPlexInterpolate(lDMBete, &lDMBeteInterp); >>>>>>> DMDestroy(&lDMBete); >>>>>>> lDMBete = lDMBeteInterp; >>>>>>> >>>>>>> DMSetUseNatural(lDMBete,PETSC_TRUE); >>>>>>> >>>>>>> PetscSF lSFMigrationSansOvl = 0; >>>>>>> PetscSF lSFMigrationOvl = 0; >>>>>>> DM lDMDistribueSansOvl = 0; >>>>>>> DM lDMAvecOverlap = 0; >>>>>>> >>>>>>> PetscPartitioner lPart; >>>>>>> DMPlexGetPartitioner(lDMBete, &lPart); >>>>>>> PetscPartitionerSetFromOptions(lPart); >>>>>>> >>>>>>> PetscSection section; >>>>>>> PetscInt numFields = 1; >>>>>>> PetscInt numBC = 0; >>>>>>> PetscInt numComp[1] = {1}; >>>>>>> PetscInt numDof[4] = {1, 0, 0, 0}; >>>>>>> PetscInt bcFields[1] = {0}; >>>>>>> IS bcPoints[1] = {NULL}; >>>>>>> >>>>>>> DMSetNumFields(lDMBete, numFields); >>>>>>> >>>>>>> DMPlexCreateSection(lDMBete, NULL, numComp, numDof, numBC, >>>>>>> bcFields, bcPoints, NULL, NULL, §ion); >>>>>>> DMSetLocalSection(lDMBete, section); >>>>>>> >>>>>>> DMPlexDistribute(lDMBete, 0, &lSFMigrationSansOvl, >>>>>>> &lDMDistribueSansOvl); // segfault! >>>>>>> >>>>>>> =========== >>>>>>> >>>>>>> So we have other question/remarks: >>>>>>> >>>>>>> 3- Maybe PETSc expect something specific that is missing/not >>>>>>> verified: for example, we didn't gave any coordinates since we just want to >>>>>>> partition and compute overlap for the mesh... and then recover our element >>>>>>> numbers in a "simple way" >>>>>>> >>>>>>> 4- We are telling ourselves it is somewhat a "big price to pay" to >>>>>>> have to build an unused section to have the global to natural ordering set >>>>>>> ? Could this requirement be avoided? >>>>>>> >>>>>> I don't think so. There would have to be _some_ way of describing >>>>>> your data layout in terms of mesh points, and I do not see how you could >>>>>> use less memory doing that. >>>>>> >>>>>>> 5- Are there any improvement towards our usages in 3.16 release? >>>>>>> >>>>>> Let me try and run the code above. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> On 2021-09-29 7:39 p.m., Matthew Knepley wrote: >>>>>>> >>>>>>> On Wed, Sep 29, 2021 at 5:18 PM Eric Chamberland < >>>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I come back with _almost_ the original question: >>>>>>>> >>>>>>>> I would like to add an integer information (*our* original element >>>>>>>> number, not petsc one) on each element of the DMPlex I create with >>>>>>>> DMPlexBuildFromCellListParallel. >>>>>>>> >>>>>>>> I would like this interger to be distribruted by or the same way >>>>>>>> DMPlexDistribute distribute the mesh. >>>>>>>> >>>>>>>> Is it possible to do this? >>>>>>>> >>>>>>> >>>>>>> I think we already have support for what you want. If you call >>>>>>> >>>>>>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >>>>>>> >>>>>>> before DMPlexDistribute(), it will compute a PetscSF encoding the >>>>>>> global to natural map. You >>>>>>> can get it with >>>>>>> >>>>>>> >>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >>>>>>> >>>>>>> and use it with >>>>>>> >>>>>>> >>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >>>>>>> >>>>>>> Is this sufficient? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> On 2021-07-14 1:18 p.m., Eric Chamberland wrote: >>>>>>>> > Hi, >>>>>>>> > >>>>>>>> > I want to use DMPlexDistribute from PETSc for computing >>>>>>>> overlapping >>>>>>>> > and play with the different partitioners supported. >>>>>>>> > >>>>>>>> > However, after calling DMPlexDistribute, I noticed the elements >>>>>>>> are >>>>>>>> > renumbered and then the original number is lost. >>>>>>>> > >>>>>>>> > What would be the best way to keep track of the element >>>>>>>> renumbering? >>>>>>>> > >>>>>>>> > a) Adding an optional parameter to let the user retrieve a vector >>>>>>>> or >>>>>>>> > "IS" giving the old number? >>>>>>>> > >>>>>>>> > b) Adding a DMLabel (seems a wrong good solution) >>>>>>>> > >>>>>>>> > c) Other idea? >>>>>>>> > >>>>>>>> > Of course, I don't want to loose performances with the need of >>>>>>>> this >>>>>>>> > "mapping"... >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > >>>>>>>> > Eric >>>>>>>> > >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Universit? Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Eric Chamberland, ing., M. Ing >>>>>>> Professionnel de recherche >>>>>>> GIREF/Universit? Laval >>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Universit? Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Universit? Laval >>>>> (418) 656-2131 poste 41 22 42 >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbnbhlbilhmjdpfg.png Type: image/png Size: 42972 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eejjfmbjimlkboec.png Type: image/png Size: 87901 bytes Desc: not available URL: From knepley at gmail.com Tue Nov 2 04:22:41 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 2 Nov 2021 05:22:41 -0400 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: On Mon, Nov 1, 2021 at 11:20 PM ?? wrote: > Dear Matthew, > > I built a test problem using the strategy you suggested. It works! It is > enough for me right now. Thank you very much. > > ! 9----------8----------13 > ! /| /| /| > ! / | / | / | > ! / | / | / | > ! 6---------7---------12 | > ! | | | | | | > ! | | | | | | > ! | | | | | | > ! | | | | | | > ! | 5------|---4-------|--11--------17------16 > ! | / | / | / / / > ! | / | / | / / / > ! |/ |/ |/ / / > ! 2-- ------3---------10--------14-------15 > > coneSize = (/ 8,8,8,8, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 /) > cones = (/ 2,5,4,3,6,7,8,9, 3,4,11,10,7,12,13,8, & > 10,14,17,11, 10,14,17,11, 14,15,16,17, 14,15,16,17 /) > > There is still a problem left, however, that is how to build a 3D cell > from a 2 vertex segment. I think I could construct a virtual tetrahedron. > > Finally, It can't be denied that this approach needs additional effort in > mesh construction (and It is something strange to construct a 3D object > from a one dimensional segment). In fact, structures with different > topological dimensions are not that rare. Parachute, for example, may be > composed with two dimensional parafoil and one dimensional cord. FRP(Fiber > reinforced plastics) may be modelled by one dimensional reinforcement and > three dimensional plastics. It is therefore, I am wondering, if PETSc would > take into consider of such cases by, for example > > - Enable a face, DM_POLYTOPE_SEGMENT2D, defined by one edge. And > - Eable cells, such > as DM_POLYTOPE_QUADRILATERAL3D, DM_POLYTOPE_TRIANGLE3D and > DM_POLYTOPE_SEGMENT3D, with one face. > I think I am still not being completely clear. These types of cells definitely exist. However, let's take the case of the parachute. The one dimensional cords are part of the two dimensional patches, which are part of a three dimensional volume. This is how we have modeled it. You assemble different equations on different pieces, but that does not affect the mesh. Thanks, Matt > Sorry for a newbie in PETSc to provide such a suggestion. But if it could > be accomplished, it would help mech structural engineering, civil > engineering, and solid mechanics applicationers. > > Best regards, > > Yuan > > > > 2021?11?1?(?) 18:32 Matthew Knepley : > >> On Sun, Oct 31, 2021 at 12:21 PM ?? wrote: >> >>> Dear Matt >>> >>> Thank you for your detailed explanation. >>> >>> First, I would like to answer your question about my application where >>> low dimensional features are not embedded in the larger volume. It is quite >>> general in structural engineering. For example, buildings are generally >>> modelled as shells and beams, which are two and one dimension respectively. >>> While building foundation is modelled by solid elements, which is >>> three dimension, at the same time. >>> >> >> I think I see what you want now. Let me make a suggestion (along the >> lines of what Mark said), and attempt to justify it by answering some >> questions. >> >> Suggestion: I think you should consider using a volumetric mesh for your >> problem >> >> Q1: Can I get the same algebraic system this way? >> >> Yes. You would only assign unknowns to faces and edges of the mesh where >> you have shell and beam elements. >> >> Q2: What are the advantages? >> >> You would get topological connectivity of all parts of the structure, >> automatic coupling of the different formulations, >> and you could separately solve the different structures using block >> preconditioners, while still forming a unified >> residual so that you can guarantee a consistent solution. >> >> Q3: Wouldn't this be expensive? >> >> No. For the shells and beams, you would still need the vertices, edges, >> and faces. First, by the Euler relation, these would >> outnumbers the additional cells. Second, since no unknowns would be >> associated with the cells, only additional memory in >> the mesh would be used, not the system solves. Memory and time are >> dominated by the solve, so this should be in the noise. >> >> What do you think? >> >> Thanks, >> >> Matt >> >> >>> Secondly, It is regrettably to find that DMComposite is not available >>> for me, because all my solid, shell, and beam elements are connected each >>> other. >>> >>> At last, I have build a simple program to see if DMPlexSetCellType() >>> works for me, following the suggestion of functions in PETSc like >>> DMPlexCreateCGNS. But it failed when it tried to do DMPlexInterpolate >>> ! 9----------8---------13 >>> ! /| /| /| >>> ! / | / | / | >>> ! / | / | / | >>> ! 6---------7---------12 | >>> ! | | | | | | >>> ! | | | | | | >>> ! | | | | | | >>> ! | | | | | | >>> ! | 5------|---4-------|-11--------17--------16 >>> ! | / | / | / / / >>> ! | / | / | / / / >>> ! |/ |/ |/ / / >>> ! 2---------3---------10--------14-------15 >>> >>> The calculation result are follows. It seems that the cell type are set >>> correctly but depth is still 2. >>> -------------------------------------------------------------------- >>> DM Object: TestMesh 2 MPI processes >>> type: plex >>> TestMesh in 3 dimensions: >>> 0-cells: 16 0 >>> 3-cells: 20 (18) 0 >>> Labels: >>> celltype: 3 strata with value/size (7 (2), 4 (2), 0 (16)) >>> depth: 2 strata with value/size (0 (16), 1 (20)) >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: Array was not checked out >>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.16.0-351-g743e004674 >>> GIT Date: 2021-10-29 09:32:23 -0500 >>> [0]PETSC ERROR: ./ex3f90 on a arch-linux-c-debug named DESKTOP-9ITFSBM >>> by hillyuan Mon Nov 1 00:26:39 2021 >>> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc >>> --with-fc=mpiifort --with-fortran-bindings=1 >>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.3.0 >>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.3.0 >>> [0]PETSC ERROR: #1 DMRestoreWorkArray() at >>> /home/hillyuan/programs/petsc/src/dm/interface/dm.c:1580 >>> [0]PETSC ERROR: #2 DMPlexRestoreRawFaces_Internal() at >>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:323 >>> [0]PETSC ERROR: #3 DMPlexInterpolateFaces_Internal() at >>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:375 >>> [0]PETSC ERROR: #4 DMPlexInterpolate() at >>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:1340 >>> >>> ----------------------------------------------------------------------------------------- >>> I attached my test program in this mail. It is much appreciated that you >>> could provide any suggestion. >>> >>> Best regards, >>> >>> Yuan >>> >>> >>> 2021?10?31?(?) 21:16 Matthew Knepley : >>> >>>> On Thu, Oct 28, 2021 at 10:48 PM ?? wrote: >>>> >>>>> Dear Matt, >>>>> >>>>> My mesh is something like the following figure, which is composed >>>>> of three elements : one hexahedron(solid element), one quadrilateral (shell >>>>> element), and one line (beam element). I found the function "TestEmptyStrata" >>>>> in file \dm\impls\plex\tests\ex11.c would be a good example to read in such >>>>> a kind of mesh by using DMPlexSetCone. But a problem is that you should >>>>> declare all faces and edges of hexahedron element, all edges in >>>>> quadrilateral element by DMPlexSetCone, otherwise PETsc could not do >>>>> topological interpolation afterwards. Am I right here? >>>>> As general in FEM mesh, my mesh does not contain any information >>>>> about faces or edges of solid elements. That's why I consider using >>>>> DMCOMPOSITE. That is >>>>> >>>>> - Put hexahedron, quadrilateral, and line elements into different >>>>> DM structures. >>>>> - do topological interpolation in those DMs separately. >>>>> - composite them. >>>>> >>>>> Is there anything wrong in my above consideration? Any suggestions? >>>>> >>>>> ------------ >>>>> /| /| >>>>> / | / | cell 0: Hex >>>>> / | / | >>>>> ------------/ | >>>>> | | | | >>>>> | | | | cell 1: Quad >>>>> | --------|---|------------ >>>>> | / | / / >>>>> | / | / / >>>>> |/ |/ / >>>>> ------------------------------------------- >>>>> cell 2: line >>>>> >>>>> Much thanks for your help. >>>>> >>>> >>>> If you are solving something where everything is embedded in a >>>> volumetric mesh, then there is no problem. However, if you really have >>>> the mesh above, where lower dimensional pieces are sticking out of the >>>> mesh, then Plex can represent the mesh, but automatic interpolation >>>> (creation of edges and faces) will not work. Why is this? We use depth >>>> in the DAG as a proxy for cell dimension, but this will no longer work >>>> if faces are not part of a volume. >>>> >>>> Will DMCOMPOSITE do what you want? It depends. It will be able to lay >>>> out a vector, but it will not know about any topological connectivity >>>> between the meshes and will not preallocate a Jacobian with any >>>> interaction. If the meshes are truly separate, this is fine. If not, it is >>>> not that >>>> useful. >>>> >>>> Could you modify the existing code to support this? Yes, it would not >>>> be terribly difficult. When you load the mesh, you must know what kind >>>> of cell you are loading. You could explicitly set this using >>>> DMPlexSetCellType(). Then, instead of taking a certain height stratum of >>>> the DAG >>>> to loop over, you would instead use all cells marked with a certain >>>> cell type. The rest of the interpolation code should work fine. >>>> >>>> What kind of physics do you have where low dimensional features are not >>>> embedded in the larger volume? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Yuan >>>>> >>>>> 2021?10?28?(?) 22:05 Matthew Knepley : >>>>> >>>>>> On Thu, Oct 28, 2021 at 4:59 AM ?? wrote: >>>>>> >>>>>>> Dear Matt, >>>>>>> >>>>>>> Thank you for your quick response. >>>>>>> >>>>>>> I think what you mean is to build DAG from my mesh at first and then >>>>>>> call DMPlexCreateFromDAG >>>>>>> () >>>>>>> to construct DMPlex. >>>>>>> >>>>>> >>>>>> No, I do not mean that. >>>>>> >>>>>> >>>>>>> A new problem is, as I know, the function DMPlexInterpolate would >>>>>>> generate points with different depth. What's the difference between those >>>>>>> faces and segment elements generated by DMPlexInterpolate with that >>>>>>> defined by the original mesh, or should we not use DMPlexInterpolate in >>>>>>> such a case? >>>>>>> >>>>>>> On the other hand, can DMComposite be used in this case by defining >>>>>>> DMPlex with different topological dimensions at first and then composite >>>>>>> them? >>>>>>> >>>>>> >>>>>> You do not need that. I am obviously not understanding your question. >>>>>> My short answer is that Plex _already_ handles cells of different >>>>>> dimension automatically without anything extra. >>>>>> >>>>>> Maybe it would help if you defined a specific problem you have. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Thanks in advance. >>>>>>> >>>>>>> Yuan >>>>>>> >>>>>>> >>>>>>> 2021?10?27?(?) 19:27 Matthew Knepley : >>>>>>> >>>>>>>> On Wed, Oct 27, 2021 at 4:50 AM ?? wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I am trying to parallelize my serial FEM program using PETSc. This >>>>>>>>> program calculates structure deformation by using various types of elements >>>>>>>>> such as solid, shell, beam, and truss. At the very beginning, I found it >>>>>>>>> was hard for me to put such kinds of elements into DMPlex. Because solid >>>>>>>>> elements are topologically three dimensional, shell element two, and beam >>>>>>>>> or truss are topologically one-dimensional elements. After reading chapter >>>>>>>>> 2.10: "DMPlex: Unstructured Grids in PETSc" of users manual carefully, I >>>>>>>>> found the provided functions, such as DMPlexSetCone, cannot declare those >>>>>>>>> topological differences. >>>>>>>>> >>>>>>>>> My question is : Is it possible and how to define all those >>>>>>>>> topologically different elements into a DMPlex struct? >>>>>>>>> >>>>>>>> >>>>>>>> Yes. The idea is to program in a dimension-independent way, so that >>>>>>>> the code can handle cells of any dimension. >>>>>>>> What you probably want is the "depth" in the DAG representation, >>>>>>>> which you can think of as the dimension of a cell. >>>>>>>> >>>>>>>> >>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetPointDepth.html#DMPlexGetPointDepth >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thanks in advance! >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Yuan. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Tue Nov 2 08:35:42 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Tue, 2 Nov 2021 08:35:42 -0500 (CDT) Subject: [petsc-users] petsc-3.16.1 now available Message-ID: Dear PETSc users, The patch release petsc-3.16.1 is now available for download. https://petsc.org/release/download/ Satish From Eric.Chamberland at giref.ulaval.ca Tue Nov 2 21:27:26 2021 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Tue, 2 Nov 2021 22:27:26 -0400 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <631832bb-4953-a3eb-13c6-0f7fe17e869d@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> Message-ID: <5751c2d4-e5d7-54d5-7f77-5bdb13c4a294@giref.ulaval.ca> Hi Matthew, yes, it all works now! I modified the example to do a NaturalToGlobal then a GlobalToNatural and test if we have the good result... The test would just need that we "force" the computed partition to a given result, then we could check the result of the NaturalToGlobal independently of the partitioner... Thanks a lot! Eric On 2021-11-02 5:16 a.m., Matthew Knepley wrote: > On Mon, Nov 1, 2021 at 5:24 PM Eric Chamberland > > wrote: > > Hi Matthew, > > ok, good news then! :) > > Meanwhile, I tested it the "other way" with > DMPlexGlobalToNaturalBegin/End and it does not give a "sensed" > result neither. > > Probably it does something like "GlobalToSequential" too? > > I am not sure how it worked before, but I think I have it fixed. I get > the output you expect for the test. I force-pushed the branch > so you will have to check it out again. > > ? Thanks! > > ? ? ?Matt > > Thanks! > > Eric > > > On 2021-11-01 3:25 p.m., Matthew Knepley wrote: >> On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland >> > > wrote: >> >> Hi Matthew, >> >> we do not know if DMPlexNaturalToGlobalBegin/End is buggy or >> if it is our comprehension of what it should do that is wrong... >> >> Would you just check if what we try to do from line 313 to >> 356 is good or wrong? >> >> The expected result is that the global vector "lGlobalVec" >> contains the element numbers from the initial partition that >> have been put into "lNatVec". >> >> Thanks a lot for any insights!! >> >> Okay, I have found the problem.I confess to not actually looking >> very closely at the implementation before. GlobalToNatural was >> assuming that >> the "natural" was also sequential, and looking at it all the >> tests are eq-to-parallel distribution. I need to fix it for a >> parallel natural state. Hopefully >> it will not take me long. >> >> ? Thanks, >> >> ? ? ?Matt >> >> Eric >> >> >> On 2021-10-27 2:32 p.m., Eric Chamberland wrote: >>> >>> Hi Matthew, >>> >>> we continued the example.? Now it must be our misuse of >>> PETSc that produced the wrong result. >>> >>> As stated into the code: >>> >>> // The call to DMPlexNaturalToGlobalBegin/End does not >>> produce our expected result... >>> ? // In lGlobalVec, we expect to have: >>> ? /* >>> ?? * Process [0] >>> ?? * 2. >>> ?? * 4. >>> ?? * 8. >>> ?? * 3. >>> ?? * 9. >>> ?? * Process [1] >>> ?? * 1. >>> ?? * 5. >>> ?? * 7. >>> ?? * 0. >>> ?? * 6. >>> ?? * >>> ?? * but we obtained: >>> ?? * >>> ?? * Process [0] >>> ?? * 2. >>> ?? * 4. >>> ?? * 8. >>> ?? * 0. >>> ?? * 0. >>> ?? * Process [1] >>> ?? * 0. >>> ?? * 0. >>> ?? * 0. >>> ?? * 0. >>> ?? * 0. >>> ?? */ >>> >>> (see attached ex44.c) >>> >>> Thanks, >>> >>> Eric >>> >>> On 2021-10-27 1:25 p.m., Eric Chamberland wrote: >>>> >>>> Great! >>>> >>>> Thanks Matthew, it is working for me up to that point! >>>> >>>> We are continuing the ex44.c and forward it to you at the >>>> next blocking point... >>>> >>>> Eric >>>> >>>> On 2021-10-27 11:14 a.m., Matthew Knepley wrote: >>>>> On Wed, Oct 27, 2021 at 8:29 AM Eric Chamberland >>>>> >>>> > wrote: >>>>> >>>>> Hi Matthew, >>>>> >>>>> the smallest mesh which crashes the code is a 2x5 mesh: >>>>> >>>>> See the modified ex44.c >>>>> >>>>> With smaller meshes(2x2, 2x4, etc), it passes...? But >>>>> it bugs latter when I try to use >>>>> DMPlexNaturalToGlobalBegin but let's keep that other >>>>> problem for later... >>>>> >>>>> Thanks a lot for helping digging into this! :) >>>>> >>>>> I have made a small fix in this branch >>>>> >>>>> https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n >>>>> >>>>> >>>>> It seems to run for me. Can you check it? >>>>> >>>>> ? Thanks, >>>>> >>>>> ? ? ?Matt >>>>> >>>>> Eric >>>>> >>>>> (sorry if you received this for a? 2nd times, I have >>>>> trouble with my mail) >>>>> >>>>> On 2021-10-26 4:35 p.m., Matthew Knepley wrote: >>>>>> On Tue, Oct 26, 2021 at 1:35 PM Eric Chamberland >>>>>> >>>>> > wrote: >>>>>> >>>>>> Here is a screenshot of the partition I hard >>>>>> coded (top) and vertices/element numbers (down): >>>>>> >>>>>> I have not yet modified the ex44.c example to >>>>>> properly assign the coordinates... >>>>>> >>>>>> (but I would not have done it like it is in the >>>>>> last version because the sCoords array is the >>>>>> global array with global vertices number) >>>>>> >>>>>> I will have time to do this tomorrow... >>>>>> >>>>>> Maybe I can first try to reproduce all this with >>>>>> a smaller mesh? >>>>>> >>>>>> >>>>>> That might make it easier to find a problem. >>>>>> >>>>>> ? Thanks! >>>>>> >>>>>> ? ? ?Matt >>>>>> >>>>>> Eric >>>>>> >>>>>> On 2021-10-26 9:46 a.m., Matthew Knepley wrote: >>>>>>> Okay, I ran it. Something seems off with the >>>>>>> mesh. First, I cannot simply explain the >>>>>>> partition. The number of shared vertices and edges >>>>>>> does not seem to come from a straight cut. >>>>>>> Second, the mesh look scrambled on output. >>>>>>> >>>>>>> ? Thanks, >>>>>>> >>>>>>> ? ? Matt >>>>>>> >>>>>>> On Sun, Oct 24, 2021 at 11:49 PM Eric >>>>>>> Chamberland >>>>>> > wrote: >>>>>>> >>>>>>> Hi Matthew, >>>>>>> >>>>>>> ok, I started back from your ex44.c example >>>>>>> and added the global array of coordinates. I >>>>>>> just have to code the creation of the local >>>>>>> coordinates now. >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> On 2021-10-20 6:55 p.m., Matthew Knepley wrote: >>>>>>>> On Wed, Oct 20, 2021 at 3:06 PM Eric >>>>>>>> Chamberland >>>>>>>> >>>>>>> > >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Matthew, >>>>>>>> >>>>>>>> we tried to reproduce the error in a >>>>>>>> simple example. >>>>>>>> >>>>>>>> The context is the following: We hard >>>>>>>> coded the mesh and initial partition >>>>>>>> into the code (see sConnectivity and >>>>>>>> sInitialPartition) for 2 ranks and try >>>>>>>> to create a section in order to use the >>>>>>>> DMPlexNaturalToGlobalBegin function to >>>>>>>> retreive our initial element numbers. >>>>>>>> >>>>>>>> Now the call to DMPlexDistribute give >>>>>>>> different errors depending on what type >>>>>>>> of component we ask the field to be >>>>>>>> created.? For our objective, we would >>>>>>>> like a global field to be created on >>>>>>>> elements only (like a P0 interpolation). >>>>>>>> >>>>>>>> We now have the following error generated: >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- >>>>>>>> Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Petsc has generated >>>>>>>> inconsistent data >>>>>>>> [0]PETSC ERROR: Inconsistency in >>>>>>>> indices, 18 should be 17 >>>>>>>> [0]PETSC ERROR: See >>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> >>>>>>>> for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version >>>>>>>> 3.15.0, Mar 30, 2021 >>>>>>>> [0]PETSC ERROR: ./bug on a? named rohan >>>>>>>> by ericc Wed Oct 20 14:52:36 2021 >>>>>>>> [0]PETSC ERROR: Configure options >>>>>>>> --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 >>>>>>>> --with-mpi-compilers=1 >>>>>>>> --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 >>>>>>>> --with-cxx-dialect=C++14 >>>>>>>> --with-make-np=12 >>>>>>>> --with-shared-libraries=1 >>>>>>>> --with-debugging=yes --with-memalign=64 >>>>>>>> --with-visibility=0 >>>>>>>> --with-64-bit-indices=0 >>>>>>>> --download-ml=yes --download-mumps=yes >>>>>>>> --download-superlu=yes >>>>>>>> --download-hpddm=yes >>>>>>>> --download-slepc=yes >>>>>>>> --download-superlu_dist=yes >>>>>>>> --download-parmetis=yes >>>>>>>> --download-ptscotch=yes >>>>>>>> --download-metis=yes >>>>>>>> --download-strumpack=yes >>>>>>>> --download-suitesparse=yes >>>>>>>> --download-hypre=yes >>>>>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>>> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>>> --with-scalapack=1 >>>>>>>> --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include >>>>>>>> --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>>> -lmkl_scalapack_lp64 >>>>>>>> -lmkl_blacs_openmpi_lp64" >>>>>>>> [0]PETSC ERROR: #1 >>>>>>>> PetscSFCreateSectionSF() at >>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 >>>>>>>> [0]PETSC ERROR: #2 >>>>>>>> DMPlexCreateGlobalToNaturalSF() at >>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 >>>>>>>> [0]PETSC ERROR: #3 DMPlexDistribute() >>>>>>>> at >>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 >>>>>>>> [0]PETSC ERROR: #4 main() at >>>>>>>> bug_section.cc:159 >>>>>>>> [0]PETSC ERROR: No PETSc Option Table >>>>>>>> entries >>>>>>>> [0]PETSC ERROR: ----------------End of >>>>>>>> Error Message -------send entire error >>>>>>>> message to petsc-maint at mcs.anl.gov >>>>>>>> ---------- >>>>>>>> >>>>>>>> Hope the attached code is >>>>>>>> self-explaining, note that to make it >>>>>>>> short, we have not included the final >>>>>>>> part of it, just the buggy part we are >>>>>>>> encountering right now... >>>>>>>> >>>>>>>> Thanks for your insights, >>>>>>>> >>>>>>>> Thanks for making the example. I tweaked it >>>>>>>> slightly. I put in a test case that just >>>>>>>> makes a parallel 7 x 10 quad mesh. This works >>>>>>>> fine. Thus I think it must be something >>>>>>>> connected with the original mesh. It is >>>>>>>> hard to get a handle on it without the >>>>>>>> coordinates. >>>>>>>> Do you think you could put the coordinate >>>>>>>> array in? I have added the code to load >>>>>>>> them (see attached file). >>>>>>>> >>>>>>>> ? Thanks, >>>>>>>> >>>>>>>> ?Matt >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> On 2021-10-06 9:23 p.m., Matthew >>>>>>>> Knepley wrote: >>>>>>>>> On Wed, Oct 6, 2021 at 5:43 PM Eric >>>>>>>>> Chamberland >>>>>>>>> >>>>>>>> > >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Matthew, >>>>>>>>> >>>>>>>>> we tried to use that. Now, we >>>>>>>>> discovered that: >>>>>>>>> >>>>>>>>> 1- even if we "ask" for sfNatural >>>>>>>>> creation with DMSetUseNatural, it >>>>>>>>> is not created because >>>>>>>>> DMPlexCreateGlobalToNaturalSF >>>>>>>>> looks for a "section": this is not >>>>>>>>> documented in DMSetUseNaturalso we >>>>>>>>> are asking ourselfs: "is this a >>>>>>>>> permanent feature or a temporary >>>>>>>>> situation?" >>>>>>>>> >>>>>>>>> I think explaining this will help >>>>>>>>> clear up a lot. >>>>>>>>> >>>>>>>>> What the Natural2Global?map does is >>>>>>>>> permute a solution vector into the >>>>>>>>> ordering that it would have had prior >>>>>>>>> to mesh distribution. >>>>>>>>> Now, in order to do this permutation, >>>>>>>>> I need to know the original (global) >>>>>>>>> data layout. If it is not specified >>>>>>>>> _before_ distribution, we >>>>>>>>> cannot build the permutation. The >>>>>>>>> section describes the data layout, so >>>>>>>>> I need it before distribution. >>>>>>>>> >>>>>>>>> I cannot think of another way that you >>>>>>>>> would implement this, but if you want >>>>>>>>> something else, let me know. >>>>>>>>> >>>>>>>>> 2- We then tried to create a >>>>>>>>> "section" in different manners: we >>>>>>>>> took the code into the example >>>>>>>>> petsc/src/dm/impls/plex/tests/ex15.c. >>>>>>>>> However, we ended up with a segfault: >>>>>>>>> >>>>>>>>> corrupted size vs. prev_size >>>>>>>>> [rohan:07297] *** Process received >>>>>>>>> signal *** >>>>>>>>> [rohan:07297] Signal: Aborted (6) >>>>>>>>> [rohan:07297] Signal code: (-6) >>>>>>>>> [rohan:07297] [ 0] >>>>>>>>> /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >>>>>>>>> [rohan:07297] [ 1] >>>>>>>>> /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >>>>>>>>> [rohan:07297] [ 2] >>>>>>>>> /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >>>>>>>>> [rohan:07297] [ 3] >>>>>>>>> /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >>>>>>>>> [rohan:07297] [ 4] >>>>>>>>> /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >>>>>>>>> [rohan:07297] [ 5] >>>>>>>>> /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >>>>>>>>> [rohan:07297] [ 6] >>>>>>>>> /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >>>>>>>>> [rohan:07297] [ 7] >>>>>>>>> /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >>>>>>>>> [rohan:07297] [ 8] >>>>>>>>> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >>>>>>>>> [rohan:07297] [ 9] >>>>>>>>> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >>>>>>>>> [rohan:07297] [10] >>>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >>>>>>>>> [rohan:07297] [11] >>>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >>>>>>>>> [rohan:07297] [12] >>>>>>>>> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >>>>>>>>> [rohan:07297] [13] >>>>>>>>> /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >>>>>>>>> >>>>>>>>> [rohan:07297] [14] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >>>>>>>>> [rohan:07297] [15] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >>>>>>>>> [rohan:07297] [16] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >>>>>>>>> [rohan:07297] [17] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >>>>>>>>> [rohan:07297] [18] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >>>>>>>>> >>>>>>>>> I am not sure what happened here, but >>>>>>>>> if you could send a sample code, I >>>>>>>>> will figure it out. >>>>>>>>> >>>>>>>>> If we do not create a section, the >>>>>>>>> call to DMPlexDistribute is >>>>>>>>> successful, but >>>>>>>>> DMPlexGetGlobalToNaturalSF return >>>>>>>>> a null SF pointer... >>>>>>>>> >>>>>>>>> Yes, it just ignores it in this case >>>>>>>>> because it does not have a global layout. >>>>>>>>> >>>>>>>>> Here are the operations we are >>>>>>>>> calling ( this is almost the code >>>>>>>>> we are using, I just removed >>>>>>>>> verifications and creation of the >>>>>>>>> connectivity which use our >>>>>>>>> parallel structure and code): >>>>>>>>> >>>>>>>>> =========== >>>>>>>>> >>>>>>>>> ? PetscInt* lCells????? = 0; >>>>>>>>> ? PetscInt lNumCorners = 0; >>>>>>>>> ? PetscInt lDimMail??? = 0; >>>>>>>>> ? PetscInt lnumCells?? = 0; >>>>>>>>> >>>>>>>>> ? //At this point we create the >>>>>>>>> cells for PETSc expected input for >>>>>>>>> DMPlexBuildFromCellListParallel >>>>>>>>> and set lNumCorners, lDimMail and >>>>>>>>> lnumCells to correct values. >>>>>>>>> ? ... >>>>>>>>> >>>>>>>>> ? DM lDMBete = 0 >>>>>>>>> DMPlexCreate(lMPIComm,&lDMBete); >>>>>>>>> >>>>>>>>> DMSetDimension(lDMBete, lDimMail); >>>>>>>>> >>>>>>>>> DMPlexBuildFromCellListParallel(lDMBete, >>>>>>>>> ????????????????????????????????? >>>>>>>>> lnumCells, >>>>>>>>> ????????????????????????????????? >>>>>>>>> PETSC_DECIDE, >>>>>>>>> pLectureElementsLocaux.reqNbTotalSommets(), >>>>>>>>> ????????????????????????????????? >>>>>>>>> lNumCorners, >>>>>>>>> ????????????????????????????????? >>>>>>>>> lCells, >>>>>>>>> ????????????????????????????????? >>>>>>>>> PETSC_NULL); >>>>>>>>> >>>>>>>>> ? DM lDMBeteInterp = 0; >>>>>>>>> DMPlexInterpolate(lDMBete, >>>>>>>>> &lDMBeteInterp); >>>>>>>>> DMDestroy(&lDMBete); >>>>>>>>> ? lDMBete = lDMBeteInterp; >>>>>>>>> >>>>>>>>> DMSetUseNatural(lDMBete,PETSC_TRUE); >>>>>>>>> >>>>>>>>> ? PetscSF lSFMigrationSansOvl = 0; >>>>>>>>> ? PetscSF lSFMigrationOvl = 0; >>>>>>>>> ? DM lDMDistribueSansOvl = 0; >>>>>>>>> ? DM lDMAvecOverlap = 0; >>>>>>>>> >>>>>>>>> PetscPartitioner lPart; >>>>>>>>> DMPlexGetPartitioner(lDMBete, &lPart); >>>>>>>>> PetscPartitionerSetFromOptions(lPart); >>>>>>>>> >>>>>>>>> PetscSection section; >>>>>>>>> PetscInt numFields?? = 1; >>>>>>>>> PetscInt numBC?????? = 0; >>>>>>>>> PetscInt numComp[1]? = {1}; >>>>>>>>> PetscInt numDof[4]?? = {1, 0, 0, 0}; >>>>>>>>> PetscInt bcFields[1] = {0}; >>>>>>>>> IS bcPoints[1] = {NULL}; >>>>>>>>> >>>>>>>>> DMSetNumFields(lDMBete, numFields); >>>>>>>>> >>>>>>>>> DMPlexCreateSection(lDMBete, NULL, >>>>>>>>> numComp, numDof, numBC, bcFields, >>>>>>>>> bcPoints, NULL, NULL, §ion); >>>>>>>>> DMSetLocalSection(lDMBete, section); >>>>>>>>> >>>>>>>>> DMPlexDistribute(lDMBete, 0, >>>>>>>>> &lSFMigrationSansOvl, >>>>>>>>> &lDMDistribueSansOvl); // segfault! >>>>>>>>> >>>>>>>>> =========== >>>>>>>>> >>>>>>>>> So we have other question/remarks: >>>>>>>>> >>>>>>>>> 3- Maybe PETSc expect something >>>>>>>>> specific that is missing/not >>>>>>>>> verified: for example, we didn't >>>>>>>>> gave any coordinates since we just >>>>>>>>> want to partition and compute >>>>>>>>> overlap for the mesh... and then >>>>>>>>> recover our element numbers in a >>>>>>>>> "simple way" >>>>>>>>> >>>>>>>>> 4- We are telling ourselves it is >>>>>>>>> somewhat a "big price to pay" to >>>>>>>>> have to build an unused section to >>>>>>>>> have the global to natural >>>>>>>>> ordering set ?? Could this >>>>>>>>> requirement be avoided? >>>>>>>>> >>>>>>>>> I don't think so. There would have to >>>>>>>>> be _some_ way of describing your data >>>>>>>>> layout in terms of mesh points, and I >>>>>>>>> do not see how you could use less >>>>>>>>> memory doing that. >>>>>>>>> >>>>>>>>> 5- Are there any improvement >>>>>>>>> towards our usages in 3.16 release? >>>>>>>>> >>>>>>>>> Let me try and run the code above. >>>>>>>>> >>>>>>>>> ? Thanks, >>>>>>>>> >>>>>>>>> ?Matt >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2021-09-29 7:39 p.m., Matthew >>>>>>>>> Knepley wrote: >>>>>>>>>> On Wed, Sep 29, 2021 at 5:18 PM >>>>>>>>>> Eric Chamberland >>>>>>>>>> >>>>>>>>> > >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I come back with _almost_ the >>>>>>>>>> original question: >>>>>>>>>> >>>>>>>>>> I would like to add an >>>>>>>>>> integer information (*our* >>>>>>>>>> original element >>>>>>>>>> number, not petsc one) on >>>>>>>>>> each element of the DMPlex I >>>>>>>>>> create with >>>>>>>>>> DMPlexBuildFromCellListParallel. >>>>>>>>>> >>>>>>>>>> I would like this interger to >>>>>>>>>> be distribruted by or the >>>>>>>>>> same way >>>>>>>>>> DMPlexDistribute distribute >>>>>>>>>> the mesh. >>>>>>>>>> >>>>>>>>>> Is it possible to do this? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think we already have support >>>>>>>>>> for what you want. If you call >>>>>>>>>> >>>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> before DMPlexDistribute(), it >>>>>>>>>> will compute a PetscSF encoding >>>>>>>>>> the global to natural map. You >>>>>>>>>> can get it with >>>>>>>>>> >>>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> and use it with >>>>>>>>>> >>>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Is this sufficient? >>>>>>>>>> >>>>>>>>>> ? Thanks, >>>>>>>>>> >>>>>>>>>> ? ? ?Matt >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> On 2021-07-14 1:18 p.m., Eric >>>>>>>>>> Chamberland wrote: >>>>>>>>>> > Hi, >>>>>>>>>> > >>>>>>>>>> > I want to use >>>>>>>>>> DMPlexDistribute from PETSc >>>>>>>>>> for computing overlapping >>>>>>>>>> > and play with the different >>>>>>>>>> partitioners supported. >>>>>>>>>> > >>>>>>>>>> > However, after calling >>>>>>>>>> DMPlexDistribute, I noticed >>>>>>>>>> the elements are >>>>>>>>>> > renumbered and then the >>>>>>>>>> original number is lost. >>>>>>>>>> > >>>>>>>>>> > What would be the best way >>>>>>>>>> to keep track of the element >>>>>>>>>> renumbering? >>>>>>>>>> > >>>>>>>>>> > a) Adding an optional >>>>>>>>>> parameter to let the user >>>>>>>>>> retrieve a vector or >>>>>>>>>> > "IS" giving the old number? >>>>>>>>>> > >>>>>>>>>> > b) Adding a DMLabel (seems >>>>>>>>>> a wrong good solution) >>>>>>>>>> > >>>>>>>>>> > c) Other idea? >>>>>>>>>> > >>>>>>>>>> > Of course, I don't want to >>>>>>>>>> loose performances with the >>>>>>>>>> need of this >>>>>>>>>> > "mapping"... >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > >>>>>>>>>> > Eric >>>>>>>>>> > >>>>>>>>>> -- >>>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>>> Professionnel de recherche >>>>>>>>>> GIREF/Universit? Laval >>>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for >>>>>>>>>> granted before they begin their >>>>>>>>>> experiments is infinitely more >>>>>>>>>> interesting than any results to >>>>>>>>>> which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Universit? Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for >>>>>>>>> granted before they begin their >>>>>>>>> experiments is infinitely more >>>>>>>>> interesting than any results to which >>>>>>>>> their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Universit? Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted >>>>>>>> before they begin their experiments is >>>>>>>> infinitely more interesting than any >>>>>>>> results to which their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Eric Chamberland, ing., M. Ing >>>>>>> Professionnel de recherche >>>>>>> GIREF/Universit? Laval >>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before >>>>>>> they begin their experiments is infinitely more >>>>>>> interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Universit? Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they >>>>>> begin their experiments is infinitely more >>>>>> interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Universit? Laval >>>>> (418) 656-2131 poste 41 22 42 >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin >>>>> their experiments is infinitely more interesting than any >>>>> results to which their experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Universit? Laval (418) 656-2131 poste 41 22 42 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbnbhlbilhmjdpfg.png Type: image/png Size: 42972 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eejjfmbjimlkboec.png Type: image/png Size: 87901 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex44.c Type: text/x-csrc Size: 13682 bytes Desc: not available URL: From yuanxi at advancesoft.jp Tue Nov 2 21:49:41 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Wed, 3 Nov 2021 11:49:41 +0900 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: Well! All objects in this world could definitely be modeled as 3D volumes. But considering costs of meshing and following calculations, in most cases it is neither practical nor necessary. And in some specific cases, e.g., to model cell membranes, 2D membrane elements may behave better than 3D volume ones. It is therefore, in many cases, modeling should be much simplified and in some cases they could be modeled as 2D or 1D ones. Following paper is obtained after just google "parachute, FSI" *A parallel 3D computational method for fluid-structure interactions in parachute systems * You can see they use cable (1D) and membrane (2D) elements to model parachutes. Those public software, such as ABAQUS, ANSYS, NASTRAN, etc all allow the mixed usage of topologically different elements. They tell the same story. It is the reality and I don't think such reality would change in the near future. Best regards Yuan 2021?11?2?(?) 18:22 Matthew Knepley : > On Mon, Nov 1, 2021 at 11:20 PM ?? wrote: > >> Dear Matthew, >> >> I built a test problem using the strategy you suggested. It works! It is >> enough for me right now. Thank you very much. >> >> ! 9----------8----------13 >> ! /| /| /| >> ! / | / | / | >> ! / | / | / | >> ! 6---------7---------12 | >> ! | | | | | | >> ! | | | | | | >> ! | | | | | | >> ! | | | | | | >> ! | 5------|---4-------|--11--------17------16 >> ! | / | / | / / / >> ! | / | / | / / / >> ! |/ |/ |/ / / >> ! 2-- ------3---------10--------14-------15 >> >> coneSize = (/ 8,8,8,8, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 /) >> cones = (/ 2,5,4,3,6,7,8,9, 3,4,11,10,7,12,13,8, & >> 10,14,17,11, 10,14,17,11, 14,15,16,17, 14,15,16,17 /) >> >> There is still a problem left, however, that is how to build a 3D cell >> from a 2 vertex segment. I think I could construct a virtual tetrahedron. >> >> Finally, It can't be denied that this approach needs additional effort in >> mesh construction (and It is something strange to construct a 3D object >> from a one dimensional segment). In fact, structures with different >> topological dimensions are not that rare. Parachute, for example, may be >> composed with two dimensional parafoil and one dimensional cord. FRP(Fiber >> reinforced plastics) may be modelled by one dimensional reinforcement and >> three dimensional plastics. It is therefore, I am wondering, if PETSc would >> take into consider of such cases by, for example >> >> - Enable a face, DM_POLYTOPE_SEGMENT2D, defined by one edge. And >> - Eable cells, such >> as DM_POLYTOPE_QUADRILATERAL3D, DM_POLYTOPE_TRIANGLE3D and >> DM_POLYTOPE_SEGMENT3D, with one face. >> > > I think I am still not being completely clear. These types of cells > definitely exist. However, let's take the case of the parachute. The one > dimensional cords are part of > the two dimensional patches, which are part of a three dimensional volume. > This is how we have modeled it. You assemble different equations on > different pieces, but > that does not affect the mesh. > > Thanks, > > Matt > > >> Sorry for a newbie in PETSc to provide such a suggestion. But if it could >> be accomplished, it would help mech structural engineering, civil >> engineering, and solid mechanics applicationers. >> >> Best regards, >> >> Yuan >> >> >> >> 2021?11?1?(?) 18:32 Matthew Knepley : >> >>> On Sun, Oct 31, 2021 at 12:21 PM ?? wrote: >>> >>>> Dear Matt >>>> >>>> Thank you for your detailed explanation. >>>> >>>> First, I would like to answer your question about my application where >>>> low dimensional features are not embedded in the larger volume. It is quite >>>> general in structural engineering. For example, buildings are generally >>>> modelled as shells and beams, which are two and one dimension respectively. >>>> While building foundation is modelled by solid elements, which is >>>> three dimension, at the same time. >>>> >>> >>> I think I see what you want now. Let me make a suggestion (along the >>> lines of what Mark said), and attempt to justify it by answering some >>> questions. >>> >>> Suggestion: I think you should consider using a volumetric mesh for your >>> problem >>> >>> Q1: Can I get the same algebraic system this way? >>> >>> Yes. You would only assign unknowns to faces and edges of the mesh where >>> you have shell and beam elements. >>> >>> Q2: What are the advantages? >>> >>> You would get topological connectivity of all parts of the structure, >>> automatic coupling of the different formulations, >>> and you could separately solve the different structures using block >>> preconditioners, while still forming a unified >>> residual so that you can guarantee a consistent solution. >>> >>> Q3: Wouldn't this be expensive? >>> >>> No. For the shells and beams, you would still need the vertices, edges, >>> and faces. First, by the Euler relation, these would >>> outnumbers the additional cells. Second, since no unknowns would be >>> associated with the cells, only additional memory in >>> the mesh would be used, not the system solves. Memory and time are >>> dominated by the solve, so this should be in the noise. >>> >>> What do you think? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Secondly, It is regrettably to find that DMComposite is not available >>>> for me, because all my solid, shell, and beam elements are connected each >>>> other. >>>> >>>> At last, I have build a simple program to see if DMPlexSetCellType() >>>> works for me, following the suggestion of functions in PETSc like >>>> DMPlexCreateCGNS. But it failed when it tried to do DMPlexInterpolate >>>> ! 9----------8---------13 >>>> ! /| /| /| >>>> ! / | / | / | >>>> ! / | / | / | >>>> ! 6---------7---------12 | >>>> ! | | | | | | >>>> ! | | | | | | >>>> ! | | | | | | >>>> ! | | | | | | >>>> ! | 5------|---4-------|-11--------17--------16 >>>> ! | / | / | / / / >>>> ! | / | / | / / / >>>> ! |/ |/ |/ / / >>>> ! 2---------3---------10--------14-------15 >>>> >>>> The calculation result are follows. It seems that the cell type are set >>>> correctly but depth is still 2. >>>> -------------------------------------------------------------------- >>>> DM Object: TestMesh 2 MPI processes >>>> type: plex >>>> TestMesh in 3 dimensions: >>>> 0-cells: 16 0 >>>> 3-cells: 20 (18) 0 >>>> Labels: >>>> celltype: 3 strata with value/size (7 (2), 4 (2), 0 (16)) >>>> depth: 2 strata with value/size (0 (16), 1 (20)) >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Array was not checked out >>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>> shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.16.0-351-g743e004674 >>>> GIT Date: 2021-10-29 09:32:23 -0500 >>>> [0]PETSC ERROR: ./ex3f90 on a arch-linux-c-debug named DESKTOP-9ITFSBM >>>> by hillyuan Mon Nov 1 00:26:39 2021 >>>> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc >>>> --with-fc=mpiifort --with-fortran-bindings=1 >>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.3.0 >>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.3.0 >>>> [0]PETSC ERROR: #1 DMRestoreWorkArray() at >>>> /home/hillyuan/programs/petsc/src/dm/interface/dm.c:1580 >>>> [0]PETSC ERROR: #2 DMPlexRestoreRawFaces_Internal() at >>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:323 >>>> [0]PETSC ERROR: #3 DMPlexInterpolateFaces_Internal() at >>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:375 >>>> [0]PETSC ERROR: #4 DMPlexInterpolate() at >>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:1340 >>>> >>>> ----------------------------------------------------------------------------------------- >>>> I attached my test program in this mail. It is much appreciated that >>>> you could provide any suggestion. >>>> >>>> Best regards, >>>> >>>> Yuan >>>> >>>> >>>> 2021?10?31?(?) 21:16 Matthew Knepley : >>>> >>>>> On Thu, Oct 28, 2021 at 10:48 PM ?? wrote: >>>>> >>>>>> Dear Matt, >>>>>> >>>>>> My mesh is something like the following figure, which is composed >>>>>> of three elements : one hexahedron(solid element), one quadrilateral (shell >>>>>> element), and one line (beam element). I found the function "TestEmptyStrata" >>>>>> in file \dm\impls\plex\tests\ex11.c would be a good example to read in such >>>>>> a kind of mesh by using DMPlexSetCone. But a problem is that you should >>>>>> declare all faces and edges of hexahedron element, all edges in >>>>>> quadrilateral element by DMPlexSetCone, otherwise PETsc could not >>>>>> do topological interpolation afterwards. Am I right here? >>>>>> As general in FEM mesh, my mesh does not contain any information >>>>>> about faces or edges of solid elements. That's why I consider using >>>>>> DMCOMPOSITE. That is >>>>>> >>>>>> - Put hexahedron, quadrilateral, and line elements into different >>>>>> DM structures. >>>>>> - do topological interpolation in those DMs separately. >>>>>> - composite them. >>>>>> >>>>>> Is there anything wrong in my above consideration? Any >>>>>> suggestions? >>>>>> >>>>>> ------------ >>>>>> /| /| >>>>>> / | / | cell 0: Hex >>>>>> / | / | >>>>>> ------------/ | >>>>>> | | | | >>>>>> | | | | cell 1: Quad >>>>>> | --------|---|------------ >>>>>> | / | / / >>>>>> | / | / / >>>>>> |/ |/ / >>>>>> ------------------------------------------- >>>>>> cell 2: line >>>>>> >>>>>> Much thanks for your help. >>>>>> >>>>> >>>>> If you are solving something where everything is embedded in a >>>>> volumetric mesh, then there is no problem. However, if you really have >>>>> the mesh above, where lower dimensional pieces are sticking out of the >>>>> mesh, then Plex can represent the mesh, but automatic interpolation >>>>> (creation of edges and faces) will not work. Why is this? We use depth >>>>> in the DAG as a proxy for cell dimension, but this will no longer work >>>>> if faces are not part of a volume. >>>>> >>>>> Will DMCOMPOSITE do what you want? It depends. It will be able to lay >>>>> out a vector, but it will not know about any topological connectivity >>>>> between the meshes and will not preallocate a Jacobian with any >>>>> interaction. If the meshes are truly separate, this is fine. If not, it is >>>>> not that >>>>> useful. >>>>> >>>>> Could you modify the existing code to support this? Yes, it would not >>>>> be terribly difficult. When you load the mesh, you must know what kind >>>>> of cell you are loading. You could explicitly set this using >>>>> DMPlexSetCellType(). Then, instead of taking a certain height stratum of >>>>> the DAG >>>>> to loop over, you would instead use all cells marked with a certain >>>>> cell type. The rest of the interpolation code should work fine. >>>>> >>>>> What kind of physics do you have where low dimensional features are >>>>> not embedded in the larger volume? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Yuan >>>>>> >>>>>> 2021?10?28?(?) 22:05 Matthew Knepley : >>>>>> >>>>>>> On Thu, Oct 28, 2021 at 4:59 AM ?? wrote: >>>>>>> >>>>>>>> Dear Matt, >>>>>>>> >>>>>>>> Thank you for your quick response. >>>>>>>> >>>>>>>> I think what you mean is to build DAG from my mesh at first and >>>>>>>> then call DMPlexCreateFromDAG >>>>>>>> () >>>>>>>> to construct DMPlex. >>>>>>>> >>>>>>> >>>>>>> No, I do not mean that. >>>>>>> >>>>>>> >>>>>>>> A new problem is, as I know, the function DMPlexInterpolate would >>>>>>>> generate points with different depth. What's the difference between those >>>>>>>> faces and segment elements generated by DMPlexInterpolate with that >>>>>>>> defined by the original mesh, or should we not use DMPlexInterpolate in >>>>>>>> such a case? >>>>>>>> >>>>>>>> On the other hand, can DMComposite be used in this case by defining >>>>>>>> DMPlex with different topological dimensions at first and then composite >>>>>>>> them? >>>>>>>> >>>>>>> >>>>>>> You do not need that. I am obviously not understanding your >>>>>>> question. My short answer is that Plex _already_ handles cells of different >>>>>>> dimension automatically without anything extra. >>>>>>> >>>>>>> Maybe it would help if you defined a specific problem you have. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Thanks in advance. >>>>>>>> >>>>>>>> Yuan >>>>>>>> >>>>>>>> >>>>>>>> 2021?10?27?(?) 19:27 Matthew Knepley : >>>>>>>> >>>>>>>>> On Wed, Oct 27, 2021 at 4:50 AM ?? wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I am trying to parallelize my serial FEM program using PETSc. >>>>>>>>>> This program calculates structure deformation by using various types of >>>>>>>>>> elements such as solid, shell, beam, and truss. At the very beginning, I >>>>>>>>>> found it was hard for me to put such kinds of elements into DMPlex. Because >>>>>>>>>> solid elements are topologically three dimensional, shell element two, and >>>>>>>>>> beam or truss are topologically one-dimensional elements. After reading >>>>>>>>>> chapter 2.10: "DMPlex: Unstructured Grids in PETSc" of users manual >>>>>>>>>> carefully, I found the provided functions, such as DMPlexSetCone, cannot >>>>>>>>>> declare those topological differences. >>>>>>>>>> >>>>>>>>>> My question is : Is it possible and how to define all those >>>>>>>>>> topologically different elements into a DMPlex struct? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yes. The idea is to program in a dimension-independent way, so >>>>>>>>> that the code can handle cells of any dimension. >>>>>>>>> What you probably want is the "depth" in the DAG representation, >>>>>>>>> which you can think of as the dimension of a cell. >>>>>>>>> >>>>>>>>> >>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetPointDepth.html#DMPlexGetPointDepth >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks in advance! >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> >>>>>>>>>> Yuan. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Tue Nov 2 23:40:25 2021 From: jed at jedbrown.org (Jed Brown) Date: Tue, 02 Nov 2021 22:40:25 -0600 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: <875yt9onra.fsf@jedbrown.org> ?? writes: > Well! All objects in this world could definitely be modeled as 3D volumes. > But considering costs of meshing and following calculations, in most cases > it is neither practical nor necessary. And in some specific cases, e.g., to > model cell membranes, 2D membrane elements may behave better than 3D volume > ones. It is therefore, in many cases, modeling should be much simplified > and in some cases they could be modeled as 2D or 1D ones. DMPlex can represent such meshes and we shouldn't hack around it by faking 3D elements (even if their cost might end up being negligible). As I understand it, the issue is that automatic interpolation doesn't support such meshes and perhaps existing readers can't create them either. This means you either need to do the leg-work up front to create these meshes or someone (perhaps you) contribute to a reader or interpolation algorithm to enable your workflow. Even if you can't work on the library code for such support, creating a merge request with a clean test would be really helpful. From nicolas.tardieu at edf.fr Wed Nov 3 03:20:21 2021 From: nicolas.tardieu at edf.fr (TARDIEU Nicolas) Date: Wed, 3 Nov 2021 08:20:21 +0000 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? Message-ID: Dear all, As a practitioner of structural mechanics, I modestly wish to support Yuan's argument. Model mixing, and consequently finite element dimension mixing, is (perhaps unfortunately) an inescapable practice of this engineering field. This has consequences not only on the nature of the meshes we use but also on the quasi-systematic recourse to linear relations (also named multipoints constraints, MPC) between the degrees of freedom of the different models (for example, linking a shell to a 3D part requires to write kinematic connections between displacements and rotations). DMPlex handling such meshes would be a really great contribution for the structural mechanics community. Best regards, Nicolas Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Nov 3 06:14:46 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Nov 2021 07:14:46 -0400 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: On Wed, Nov 3, 2021 at 4:20 AM TARDIEU Nicolas via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear all, > > As a practitioner of structural mechanics, I modestly wish to support > Yuan's argument. Model mixing, and consequently finite element dimension > mixing, is (perhaps unfortunately) an inescapable practice of this > engineering field. This has consequences not only on the nature of the > meshes we use but also on the quasi-systematic recourse to linear relations > (also named multipoints constraints, MPC) between the degrees of freedom of > the different models (for example, linking a shell to a 3D part requires to > write kinematic connections between displacements and rotations). > DMPlex handling such meshes would be a really great contribution for the > structural mechanics community. > Nicolas, >From my point of view, this is supported, so it would be good to understand what you do not think is supported: 1) Multiple models This is supported in the sense that you can use any data layout, and assemble any system you want into it. You can use beams along edges and shells on faces if you want. I have examples of different equations in different pieces of the domain. 2) Mesh with pieces hanging off This is supported in the sense that you can input such a mesh. You have to describe it completely, but it can be input and iterated over. 3) Mesh Interpolation If you only want to input vertices _and_ you want to input different dimensional objects, you will have to write code. There is no way to guess what you mean from a collection of vertices. You could a) Directly call GetRawFaces() to do the interpolation per cell before input (seems easier) b) Rewrite the interpolation loop to check whether cell types have been set and just call the appropriate interpolation if they have I guess you are arguing that I should add the code for 3). Most people I have interacted with use a mesh generator to make the meshes, so I concentrated on importing those formats. Can meshes like these be made with those generators? Thanks, Matt > Best regards, > Nicolas > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de > le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Nov 3 06:19:36 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Nov 2021 07:19:36 -0400 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: On Tue, Nov 2, 2021 at 10:49 PM ?? wrote: > Well! All objects in this world could definitely be modeled as 3D volumes. > But considering costs of meshing and following calculations, in most cases > it is neither practical nor necessary. And in some specific cases, e.g., to > model cell membranes, 2D membrane elements may behave better than 3D volume > ones. It is therefore, in many cases, modeling should be much simplified > and in some cases they could be modeled as 2D or 1D ones. > I have _never_ suggested using 3D elements, as you can see from my reply to Nicolas. I said you can use _exactly_ the element you want. I am not sure how to be more clear. Matt > Following paper is obtained after just google "parachute, FSI" > *A parallel 3D computational method for fluid-structure interactions in > parachute systems > * > > You can see they use cable (1D) and membrane (2D) elements to model > parachutes. Those public software, such as ABAQUS, ANSYS, NASTRAN, etc all > allow the mixed usage of topologically different elements. They tell the > same story. It is the reality and I don't think such reality would change > in the near future. > > Best regards > > Yuan > > 2021?11?2?(?) 18:22 Matthew Knepley : > >> On Mon, Nov 1, 2021 at 11:20 PM ?? wrote: >> >>> Dear Matthew, >>> >>> I built a test problem using the strategy you suggested. It works! It is >>> enough for me right now. Thank you very much. >>> >>> ! 9----------8----------13 >>> ! /| /| /| >>> ! / | / | / | >>> ! / | / | / | >>> ! 6---------7---------12 | >>> ! | | | | | | >>> ! | | | | | | >>> ! | | | | | | >>> ! | | | | | | >>> ! | 5------|---4-------|--11--------17------16 >>> ! | / | / | / / / >>> ! | / | / | / / / >>> ! |/ |/ |/ / / >>> ! 2-- ------3---------10--------14-------15 >>> >>> coneSize = (/ 8,8,8,8, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 /) >>> cones = (/ 2,5,4,3,6,7,8,9, 3,4,11,10,7,12,13,8, & >>> 10,14,17,11, 10,14,17,11, 14,15,16,17, 14,15,16,17 /) >>> >>> There is still a problem left, however, that is how to build a 3D cell >>> from a 2 vertex segment. I think I could construct a virtual tetrahedron. >>> >>> Finally, It can't be denied that this approach needs additional effort >>> in mesh construction (and It is something strange to construct a 3D object >>> from a one dimensional segment). In fact, structures with different >>> topological dimensions are not that rare. Parachute, for example, may be >>> composed with two dimensional parafoil and one dimensional cord. FRP(Fiber >>> reinforced plastics) may be modelled by one dimensional reinforcement and >>> three dimensional plastics. It is therefore, I am wondering, if PETSc would >>> take into consider of such cases by, for example >>> >>> - Enable a face, DM_POLYTOPE_SEGMENT2D, defined by one edge. And >>> - Eable cells, such >>> as DM_POLYTOPE_QUADRILATERAL3D, DM_POLYTOPE_TRIANGLE3D and >>> DM_POLYTOPE_SEGMENT3D, with one face. >>> >> >> I think I am still not being completely clear. These types of cells >> definitely exist. However, let's take the case of the parachute. The one >> dimensional cords are part of >> the two dimensional patches, which are part of a three dimensional >> volume. This is how we have modeled it. You assemble different equations on >> different pieces, but >> that does not affect the mesh. >> >> Thanks, >> >> Matt >> >> >>> Sorry for a newbie in PETSc to provide such a suggestion. But if it >>> could be accomplished, it would help mech structural engineering, civil >>> engineering, and solid mechanics applicationers. >>> >>> Best regards, >>> >>> Yuan >>> >>> >>> >>> 2021?11?1?(?) 18:32 Matthew Knepley : >>> >>>> On Sun, Oct 31, 2021 at 12:21 PM ?? wrote: >>>> >>>>> Dear Matt >>>>> >>>>> Thank you for your detailed explanation. >>>>> >>>>> First, I would like to answer your question about my application where >>>>> low dimensional features are not embedded in the larger volume. It is quite >>>>> general in structural engineering. For example, buildings are generally >>>>> modelled as shells and beams, which are two and one dimension respectively. >>>>> While building foundation is modelled by solid elements, which is >>>>> three dimension, at the same time. >>>>> >>>> >>>> I think I see what you want now. Let me make a suggestion (along the >>>> lines of what Mark said), and attempt to justify it by answering some >>>> questions. >>>> >>>> Suggestion: I think you should consider using a volumetric mesh for >>>> your problem >>>> >>>> Q1: Can I get the same algebraic system this way? >>>> >>>> Yes. You would only assign unknowns to faces and edges of the mesh >>>> where you have shell and beam elements. >>>> >>>> Q2: What are the advantages? >>>> >>>> You would get topological connectivity of all parts of the structure, >>>> automatic coupling of the different formulations, >>>> and you could separately solve the different structures using block >>>> preconditioners, while still forming a unified >>>> residual so that you can guarantee a consistent solution. >>>> >>>> Q3: Wouldn't this be expensive? >>>> >>>> No. For the shells and beams, you would still need the vertices, edges, >>>> and faces. First, by the Euler relation, these would >>>> outnumbers the additional cells. Second, since no unknowns would be >>>> associated with the cells, only additional memory in >>>> the mesh would be used, not the system solves. Memory and time are >>>> dominated by the solve, so this should be in the noise. >>>> >>>> What do you think? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Secondly, It is regrettably to find that DMComposite is not available >>>>> for me, because all my solid, shell, and beam elements are connected each >>>>> other. >>>>> >>>>> At last, I have build a simple program to see if DMPlexSetCellType() >>>>> works for me, following the suggestion of functions in PETSc like >>>>> DMPlexCreateCGNS. But it failed when it tried to do DMPlexInterpolate >>>>> ! 9----------8---------13 >>>>> ! /| /| /| >>>>> ! / | / | / | >>>>> ! / | / | / | >>>>> ! 6---------7---------12 | >>>>> ! | | | | | | >>>>> ! | | | | | | >>>>> ! | | | | | | >>>>> ! | | | | | | >>>>> ! | 5------|---4-------|-11--------17--------16 >>>>> ! | / | / | / / / >>>>> ! | / | / | / / / >>>>> ! |/ |/ |/ / / >>>>> ! 2---------3---------10--------14-------15 >>>>> >>>>> The calculation result are follows. It seems that the cell type are >>>>> set correctly but depth is still 2. >>>>> -------------------------------------------------------------------- >>>>> DM Object: TestMesh 2 MPI processes >>>>> type: plex >>>>> TestMesh in 3 dimensions: >>>>> 0-cells: 16 0 >>>>> 3-cells: 20 (18) 0 >>>>> Labels: >>>>> celltype: 3 strata with value/size (7 (2), 4 (2), 0 (16)) >>>>> depth: 2 strata with value/size (0 (16), 1 (20)) >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Object is in wrong state >>>>> [0]PETSC ERROR: Array was not checked out >>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>>> shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>> v3.16.0-351-g743e004674 GIT Date: 2021-10-29 09:32:23 -0500 >>>>> [0]PETSC ERROR: ./ex3f90 on a arch-linux-c-debug named DESKTOP-9ITFSBM >>>>> by hillyuan Mon Nov 1 00:26:39 2021 >>>>> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc >>>>> --with-fc=mpiifort --with-fortran-bindings=1 >>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.3.0 >>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.3.0 >>>>> [0]PETSC ERROR: #1 DMRestoreWorkArray() at >>>>> /home/hillyuan/programs/petsc/src/dm/interface/dm.c:1580 >>>>> [0]PETSC ERROR: #2 DMPlexRestoreRawFaces_Internal() at >>>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:323 >>>>> [0]PETSC ERROR: #3 DMPlexInterpolateFaces_Internal() at >>>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:375 >>>>> [0]PETSC ERROR: #4 DMPlexInterpolate() at >>>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:1340 >>>>> >>>>> ----------------------------------------------------------------------------------------- >>>>> I attached my test program in this mail. It is much appreciated that >>>>> you could provide any suggestion. >>>>> >>>>> Best regards, >>>>> >>>>> Yuan >>>>> >>>>> >>>>> 2021?10?31?(?) 21:16 Matthew Knepley : >>>>> >>>>>> On Thu, Oct 28, 2021 at 10:48 PM ?? wrote: >>>>>> >>>>>>> Dear Matt, >>>>>>> >>>>>>> My mesh is something like the following figure, which is >>>>>>> composed of three elements : one hexahedron(solid element), one >>>>>>> quadrilateral (shell element), and one line (beam element). I found the >>>>>>> function "TestEmptyStrata" in file \dm\impls\plex\tests\ex11.c >>>>>>> would be a good example to read in such a kind of mesh by using >>>>>>> DMPlexSetCone. But a problem is that you should declare all faces and edges >>>>>>> of hexahedron element, all edges in quadrilateral element by >>>>>>> DMPlexSetCone, otherwise PETsc could not do >>>>>>> topological interpolation afterwards. Am I right here? >>>>>>> As general in FEM mesh, my mesh does not contain any information >>>>>>> about faces or edges of solid elements. That's why I consider using >>>>>>> DMCOMPOSITE. That is >>>>>>> >>>>>>> - Put hexahedron, quadrilateral, and line elements into >>>>>>> different DM structures. >>>>>>> - do topological interpolation in those DMs separately. >>>>>>> - composite them. >>>>>>> >>>>>>> Is there anything wrong in my above consideration? Any >>>>>>> suggestions? >>>>>>> >>>>>>> ------------ >>>>>>> /| /| >>>>>>> / | / | cell 0: Hex >>>>>>> / | / | >>>>>>> ------------/ | >>>>>>> | | | | >>>>>>> | | | | cell 1: Quad >>>>>>> | --------|---|------------ >>>>>>> | / | / / >>>>>>> | / | / / >>>>>>> |/ |/ / >>>>>>> ------------------------------------------- >>>>>>> cell 2: line >>>>>>> >>>>>>> Much thanks for your help. >>>>>>> >>>>>> >>>>>> If you are solving something where everything is embedded in a >>>>>> volumetric mesh, then there is no problem. However, if you really have >>>>>> the mesh above, where lower dimensional pieces are sticking out of >>>>>> the mesh, then Plex can represent the mesh, but automatic interpolation >>>>>> (creation of edges and faces) will not work. Why is this? We use >>>>>> depth in the DAG as a proxy for cell dimension, but this will no longer work >>>>>> if faces are not part of a volume. >>>>>> >>>>>> Will DMCOMPOSITE do what you want? It depends. It will be able to lay >>>>>> out a vector, but it will not know about any topological connectivity >>>>>> between the meshes and will not preallocate a Jacobian with any >>>>>> interaction. If the meshes are truly separate, this is fine. If not, it is >>>>>> not that >>>>>> useful. >>>>>> >>>>>> Could you modify the existing code to support this? Yes, it would not >>>>>> be terribly difficult. When you load the mesh, you must know what kind >>>>>> of cell you are loading. You could explicitly set this using >>>>>> DMPlexSetCellType(). Then, instead of taking a certain height stratum of >>>>>> the DAG >>>>>> to loop over, you would instead use all cells marked with a certain >>>>>> cell type. The rest of the interpolation code should work fine. >>>>>> >>>>>> What kind of physics do you have where low dimensional features are >>>>>> not embedded in the larger volume? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> Yuan >>>>>>> >>>>>>> 2021?10?28?(?) 22:05 Matthew Knepley : >>>>>>> >>>>>>>> On Thu, Oct 28, 2021 at 4:59 AM ?? wrote: >>>>>>>> >>>>>>>>> Dear Matt, >>>>>>>>> >>>>>>>>> Thank you for your quick response. >>>>>>>>> >>>>>>>>> I think what you mean is to build DAG from my mesh at first and >>>>>>>>> then call DMPlexCreateFromDAG >>>>>>>>> () >>>>>>>>> to construct DMPlex. >>>>>>>>> >>>>>>>> >>>>>>>> No, I do not mean that. >>>>>>>> >>>>>>>> >>>>>>>>> A new problem is, as I know, the function DMPlexInterpolate would >>>>>>>>> generate points with different depth. What's the difference between those >>>>>>>>> faces and segment elements generated by DMPlexInterpolate with that >>>>>>>>> defined by the original mesh, or should we not use DMPlexInterpolate in >>>>>>>>> such a case? >>>>>>>>> >>>>>>>>> On the other hand, can DMComposite be used in this case by >>>>>>>>> defining DMPlex with different topological dimensions at first and then >>>>>>>>> composite them? >>>>>>>>> >>>>>>>> >>>>>>>> You do not need that. I am obviously not understanding your >>>>>>>> question. My short answer is that Plex _already_ handles cells of different >>>>>>>> dimension automatically without anything extra. >>>>>>>> >>>>>>>> Maybe it would help if you defined a specific problem you have. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thanks in advance. >>>>>>>>> >>>>>>>>> Yuan >>>>>>>>> >>>>>>>>> >>>>>>>>> 2021?10?27?(?) 19:27 Matthew Knepley : >>>>>>>>> >>>>>>>>>> On Wed, Oct 27, 2021 at 4:50 AM ?? wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I am trying to parallelize my serial FEM program using PETSc. >>>>>>>>>>> This program calculates structure deformation by using various types of >>>>>>>>>>> elements such as solid, shell, beam, and truss. At the very beginning, I >>>>>>>>>>> found it was hard for me to put such kinds of elements into DMPlex. Because >>>>>>>>>>> solid elements are topologically three dimensional, shell element two, and >>>>>>>>>>> beam or truss are topologically one-dimensional elements. After reading >>>>>>>>>>> chapter 2.10: "DMPlex: Unstructured Grids in PETSc" of users manual >>>>>>>>>>> carefully, I found the provided functions, such as DMPlexSetCone, cannot >>>>>>>>>>> declare those topological differences. >>>>>>>>>>> >>>>>>>>>>> My question is : Is it possible and how to define all those >>>>>>>>>>> topologically different elements into a DMPlex struct? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Yes. The idea is to program in a dimension-independent way, so >>>>>>>>>> that the code can handle cells of any dimension. >>>>>>>>>> What you probably want is the "depth" in the DAG representation, >>>>>>>>>> which you can think of as the dimension of a cell. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetPointDepth.html#DMPlexGetPointDepth >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks in advance! >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> >>>>>>>>>>> Yuan. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Nov 3 06:41:21 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Nov 2021 07:41:21 -0400 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: <5751c2d4-e5d7-54d5-7f77-5bdb13c4a294@giref.ulaval.ca> References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <631832bb-4953-a3eb-13c6-0f7fe17e869d@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> <5751c2d4-e5d7-54d5-7f77-5bdb13c4a294@giref.ulaval.ca> Message-ID: On Tue, Nov 2, 2021 at 10:27 PM Eric Chamberland < Eric.Chamberland at giref.ulaval.ca> wrote: > Hi Matthew, > > yes, it all works now! > > I modified the example to do a NaturalToGlobal then a GlobalToNatural and > test if we have the good result... > > The test would just need that we "force" the computed partition to a given > result, then we could check the result of the NaturalToGlobal independently > of the partitioner... > > Thanks a lot! > > No problem. Thanks for generating a great test. I will get this into the CI tonight. Thanks, Matt > Eric > > > On 2021-11-02 5:16 a.m., Matthew Knepley wrote: > > On Mon, Nov 1, 2021 at 5:24 PM Eric Chamberland < > Eric.Chamberland at giref.ulaval.ca> wrote: > >> Hi Matthew, >> >> ok, good news then! :) >> >> Meanwhile, I tested it the "other way" with >> DMPlexGlobalToNaturalBegin/End and it does not give a "sensed" result >> neither. >> >> Probably it does something like "GlobalToSequential" too? >> > I am not sure how it worked before, but I think I have it fixed. I get the > output you expect for the test. I force-pushed the branch > so you will have to check it out again. > > Thanks! > > Matt > >> Thanks! >> >> Eric >> >> >> On 2021-11-01 3:25 p.m., Matthew Knepley wrote: >> >> On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland < >> Eric.Chamberland at giref.ulaval.ca> wrote: >> >>> Hi Matthew, >>> >>> we do not know if DMPlexNaturalToGlobalBegin/End is buggy or if it is >>> our comprehension of what it should do that is wrong... >>> >>> Would you just check if what we try to do from line 313 to 356 is good >>> or wrong? >>> >>> The expected result is that the global vector "lGlobalVec" contains the >>> element numbers from the initial partition that have been put into >>> "lNatVec". >>> >>> Thanks a lot for any insights!! >>> >> Okay, I have found the problem.I confess to not actually looking very >> closely at the implementation before. GlobalToNatural was assuming that >> the "natural" was also sequential, and looking at it all the tests are >> eq-to-parallel distribution. I need to fix it for a parallel natural state. >> Hopefully >> it will not take me long. >> >> Thanks, >> >> Matt >> >>> Eric >>> >>> >>> On 2021-10-27 2:32 p.m., Eric Chamberland wrote: >>> >>> Hi Matthew, >>> >>> we continued the example. Now it must be our misuse of PETSc that >>> produced the wrong result. >>> >>> As stated into the code: >>> >>> // The call to DMPlexNaturalToGlobalBegin/End does not produce our >>> expected result... >>> // In lGlobalVec, we expect to have: >>> /* >>> * Process [0] >>> * 2. >>> * 4. >>> * 8. >>> * 3. >>> * 9. >>> * Process [1] >>> * 1. >>> * 5. >>> * 7. >>> * 0. >>> * 6. >>> * >>> * but we obtained: >>> * >>> * Process [0] >>> * 2. >>> * 4. >>> * 8. >>> * 0. >>> * 0. >>> * Process [1] >>> * 0. >>> * 0. >>> * 0. >>> * 0. >>> * 0. >>> */ >>> >>> (see attached ex44.c) >>> >>> Thanks, >>> >>> Eric >>> On 2021-10-27 1:25 p.m., Eric Chamberland wrote: >>> >>> Great! >>> >>> Thanks Matthew, it is working for me up to that point! >>> >>> We are continuing the ex44.c and forward it to you at the next blocking >>> point... >>> >>> Eric >>> On 2021-10-27 11:14 a.m., Matthew Knepley wrote: >>> >>> On Wed, Oct 27, 2021 at 8:29 AM Eric Chamberland < >>> Eric.Chamberland at giref.ulaval.ca> wrote: >>> >>>> Hi Matthew, >>>> >>>> the smallest mesh which crashes the code is a 2x5 mesh: >>>> >>>> See the modified ex44.c >>>> >>>> With smaller meshes(2x2, 2x4, etc), it passes... But it bugs latter >>>> when I try to use DMPlexNaturalToGlobalBegin but let's keep that other >>>> problem for later... >>>> >>>> Thanks a lot for helping digging into this! :) >>>> >>> I have made a small fix in this branch >>> >>> https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n >>> >>> It seems to run for me. Can you check it? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Eric >>>> >>>> (sorry if you received this for a 2nd times, I have trouble with my >>>> mail) >>>> On 2021-10-26 4:35 p.m., Matthew Knepley wrote: >>>> >>>> On Tue, Oct 26, 2021 at 1:35 PM Eric Chamberland < >>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>> >>>>> Here is a screenshot of the partition I hard coded (top) and >>>>> vertices/element numbers (down): >>>>> >>>>> I have not yet modified the ex44.c example to properly assign the >>>>> coordinates... >>>>> >>>>> (but I would not have done it like it is in the last version because >>>>> the sCoords array is the global array with global vertices number) >>>>> >>>>> I will have time to do this tomorrow... >>>>> >>>>> Maybe I can first try to reproduce all this with a smaller mesh? >>>>> >>>> >>>> That might make it easier to find a problem. >>>> >>>> Thanks! >>>> >>>> Matt >>>> >>>> >>>>> Eric >>>>> On 2021-10-26 9:46 a.m., Matthew Knepley wrote: >>>>> >>>>> Okay, I ran it. Something seems off with the mesh. First, I cannot >>>>> simply explain the partition. The number of shared vertices and edges >>>>> does not seem to come from a straight cut. Second, the mesh look >>>>> scrambled on output. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> On Sun, Oct 24, 2021 at 11:49 PM Eric Chamberland < >>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>> >>>>>> Hi Matthew, >>>>>> >>>>>> ok, I started back from your ex44.c example and added the global >>>>>> array of coordinates. I just have to code the creation of the local >>>>>> coordinates now. >>>>>> >>>>>> Eric >>>>>> On 2021-10-20 6:55 p.m., Matthew Knepley wrote: >>>>>> >>>>>> On Wed, Oct 20, 2021 at 3:06 PM Eric Chamberland < >>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>> >>>>>>> Hi Matthew, >>>>>>> >>>>>>> we tried to reproduce the error in a simple example. >>>>>>> >>>>>>> The context is the following: We hard coded the mesh and initial >>>>>>> partition into the code (see sConnectivity and sInitialPartition) for 2 >>>>>>> ranks and try to create a section in order to use the >>>>>>> DMPlexNaturalToGlobalBegin function to retreive our initial element numbers. >>>>>>> >>>>>>> Now the call to DMPlexDistribute give different errors depending on >>>>>>> what type of component we ask the field to be created. For our objective, >>>>>>> we would like a global field to be created on elements only (like a P0 >>>>>>> interpolation). >>>>>>> >>>>>>> We now have the following error generated: >>>>>>> >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>> [0]PETSC ERROR: Inconsistency in indices, 18 should be 17 >>>>>>> [0]PETSC ERROR: See >>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>> shooting. >>>>>>> [0]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021 >>>>>>> [0]PETSC ERROR: ./bug on a named rohan by ericc Wed Oct 20 14:52:36 >>>>>>> 2021 >>>>>>> [0]PETSC ERROR: Configure options >>>>>>> --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 --with-mpi-compilers=1 >>>>>>> --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 --with-cxx-dialect=C++14 >>>>>>> --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes >>>>>>> --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 >>>>>>> --download-ml=yes --download-mumps=yes --download-superlu=yes >>>>>>> --download-hpddm=yes --download-slepc=yes --download-superlu_dist=yes >>>>>>> --download-parmetis=yes --download-ptscotch=yes --download-metis=yes >>>>>>> --download-strumpack=yes --download-suitesparse=yes --download-hypre=yes >>>>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>> --with-scalapack=1 >>>>>>> --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include >>>>>>> --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>> -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" >>>>>>> [0]PETSC ERROR: #1 PetscSFCreateSectionSF() at >>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 >>>>>>> [0]PETSC ERROR: #2 DMPlexCreateGlobalToNaturalSF() at >>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 >>>>>>> [0]PETSC ERROR: #3 DMPlexDistribute() at >>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 >>>>>>> [0]PETSC ERROR: #4 main() at bug_section.cc:159 >>>>>>> [0]PETSC ERROR: No PETSc Option Table entries >>>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>>>> entire error message to petsc-maint at mcs.anl.gov---------- >>>>>>> >>>>>>> Hope the attached code is self-explaining, note that to make it >>>>>>> short, we have not included the final part of it, just the buggy part we >>>>>>> are encountering right now... >>>>>>> >>>>>>> Thanks for your insights, >>>>>>> >>>>>> Thanks for making the example. I tweaked it slightly. I put in a test >>>>>> case that just makes a parallel 7 x 10 quad mesh. This works >>>>>> fine. Thus I think it must be something connected with the original >>>>>> mesh. It is hard to get a handle on it without the coordinates. >>>>>> Do you think you could put the coordinate array in? I have added the >>>>>> code to load them (see attached file). >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>>> Eric >>>>>>> On 2021-10-06 9:23 p.m., Matthew Knepley wrote: >>>>>>> >>>>>>> On Wed, Oct 6, 2021 at 5:43 PM Eric Chamberland < >>>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>>> >>>>>>>> Hi Matthew, >>>>>>>> >>>>>>>> we tried to use that. Now, we discovered that: >>>>>>>> >>>>>>>> 1- even if we "ask" for sfNatural creation with DMSetUseNatural, it >>>>>>>> is not created because DMPlexCreateGlobalToNaturalSF looks for a "section": >>>>>>>> this is not documented in DMSetUseNaturalso we are asking ourselfs: "is >>>>>>>> this a permanent feature or a temporary situation?" >>>>>>>> >>>>>>> I think explaining this will help clear up a lot. >>>>>>> >>>>>>> What the Natural2Global map does is permute a solution vector into >>>>>>> the ordering that it would have had prior to mesh distribution. >>>>>>> Now, in order to do this permutation, I need to know the original >>>>>>> (global) data layout. If it is not specified _before_ distribution, we >>>>>>> cannot build the permutation. The section describes the data >>>>>>> layout, so I need it before distribution. >>>>>>> >>>>>>> I cannot think of another way that you would implement this, but if >>>>>>> you want something else, let me know. >>>>>>> >>>>>>>> 2- We then tried to create a "section" in different manners: we >>>>>>>> took the code into the example petsc/src/dm/impls/plex/tests/ex15.c. >>>>>>>> However, we ended up with a segfault: >>>>>>>> >>>>>>>> corrupted size vs. prev_size >>>>>>>> [rohan:07297] *** Process received signal *** >>>>>>>> [rohan:07297] Signal: Aborted (6) >>>>>>>> [rohan:07297] Signal code: (-6) >>>>>>>> [rohan:07297] [ 0] /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >>>>>>>> [rohan:07297] [ 1] /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >>>>>>>> [rohan:07297] [ 2] /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >>>>>>>> [rohan:07297] [ 3] /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >>>>>>>> [rohan:07297] [ 4] /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >>>>>>>> [rohan:07297] [ 5] /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >>>>>>>> [rohan:07297] [ 6] /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >>>>>>>> [rohan:07297] [ 7] /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >>>>>>>> [rohan:07297] [ 8] >>>>>>>> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >>>>>>>> [rohan:07297] [ 9] >>>>>>>> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >>>>>>>> [rohan:07297] [10] >>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >>>>>>>> [rohan:07297] [11] >>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >>>>>>>> [rohan:07297] [12] >>>>>>>> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >>>>>>>> [rohan:07297] [13] /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >>>>>>>> >>>>>>>> [rohan:07297] [14] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >>>>>>>> [rohan:07297] [15] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >>>>>>>> [rohan:07297] [16] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >>>>>>>> [rohan:07297] [17] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >>>>>>>> [rohan:07297] [18] >>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >>>>>>>> >>>>>>> I am not sure what happened here, but if you could send a sample >>>>>>> code, I will figure it out. >>>>>>> >>>>>>>> If we do not create a section, the call to DMPlexDistribute is >>>>>>>> successful, but DMPlexGetGlobalToNaturalSF return a null SF pointer... >>>>>>>> >>>>>>> Yes, it just ignores it in this case because it does not have a >>>>>>> global layout. >>>>>>> >>>>>>>> Here are the operations we are calling ( this is almost the code we >>>>>>>> are using, I just removed verifications and creation of the connectivity >>>>>>>> which use our parallel structure and code): >>>>>>>> >>>>>>>> =========== >>>>>>>> >>>>>>>> PetscInt* lCells = 0; >>>>>>>> PetscInt lNumCorners = 0; >>>>>>>> PetscInt lDimMail = 0; >>>>>>>> PetscInt lnumCells = 0; >>>>>>>> >>>>>>>> //At this point we create the cells for PETSc expected input for >>>>>>>> DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and lnumCells >>>>>>>> to correct values. >>>>>>>> ... >>>>>>>> >>>>>>>> DM lDMBete = 0 >>>>>>>> DMPlexCreate(lMPIComm,&lDMBete); >>>>>>>> >>>>>>>> DMSetDimension(lDMBete, lDimMail); >>>>>>>> >>>>>>>> DMPlexBuildFromCellListParallel(lDMBete, >>>>>>>> lnumCells, >>>>>>>> PETSC_DECIDE, >>>>>>>> >>>>>>>> pLectureElementsLocaux.reqNbTotalSommets(), >>>>>>>> lNumCorners, >>>>>>>> lCells, >>>>>>>> PETSC_NULL); >>>>>>>> >>>>>>>> DM lDMBeteInterp = 0; >>>>>>>> DMPlexInterpolate(lDMBete, &lDMBeteInterp); >>>>>>>> DMDestroy(&lDMBete); >>>>>>>> lDMBete = lDMBeteInterp; >>>>>>>> >>>>>>>> DMSetUseNatural(lDMBete,PETSC_TRUE); >>>>>>>> >>>>>>>> PetscSF lSFMigrationSansOvl = 0; >>>>>>>> PetscSF lSFMigrationOvl = 0; >>>>>>>> DM lDMDistribueSansOvl = 0; >>>>>>>> DM lDMAvecOverlap = 0; >>>>>>>> >>>>>>>> PetscPartitioner lPart; >>>>>>>> DMPlexGetPartitioner(lDMBete, &lPart); >>>>>>>> PetscPartitionerSetFromOptions(lPart); >>>>>>>> >>>>>>>> PetscSection section; >>>>>>>> PetscInt numFields = 1; >>>>>>>> PetscInt numBC = 0; >>>>>>>> PetscInt numComp[1] = {1}; >>>>>>>> PetscInt numDof[4] = {1, 0, 0, 0}; >>>>>>>> PetscInt bcFields[1] = {0}; >>>>>>>> IS bcPoints[1] = {NULL}; >>>>>>>> >>>>>>>> DMSetNumFields(lDMBete, numFields); >>>>>>>> >>>>>>>> DMPlexCreateSection(lDMBete, NULL, numComp, numDof, numBC, >>>>>>>> bcFields, bcPoints, NULL, NULL, §ion); >>>>>>>> DMSetLocalSection(lDMBete, section); >>>>>>>> >>>>>>>> DMPlexDistribute(lDMBete, 0, &lSFMigrationSansOvl, >>>>>>>> &lDMDistribueSansOvl); // segfault! >>>>>>>> >>>>>>>> =========== >>>>>>>> >>>>>>>> So we have other question/remarks: >>>>>>>> >>>>>>>> 3- Maybe PETSc expect something specific that is missing/not >>>>>>>> verified: for example, we didn't gave any coordinates since we just want to >>>>>>>> partition and compute overlap for the mesh... and then recover our element >>>>>>>> numbers in a "simple way" >>>>>>>> >>>>>>>> 4- We are telling ourselves it is somewhat a "big price to pay" to >>>>>>>> have to build an unused section to have the global to natural ordering set >>>>>>>> ? Could this requirement be avoided? >>>>>>>> >>>>>>> I don't think so. There would have to be _some_ way of describing >>>>>>> your data layout in terms of mesh points, and I do not see how you could >>>>>>> use less memory doing that. >>>>>>> >>>>>>>> 5- Are there any improvement towards our usages in 3.16 release? >>>>>>>> >>>>>>> Let me try and run the code above. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> >>>>>>>> On 2021-09-29 7:39 p.m., Matthew Knepley wrote: >>>>>>>> >>>>>>>> On Wed, Sep 29, 2021 at 5:18 PM Eric Chamberland < >>>>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I come back with _almost_ the original question: >>>>>>>>> >>>>>>>>> I would like to add an integer information (*our* original element >>>>>>>>> number, not petsc one) on each element of the DMPlex I create with >>>>>>>>> DMPlexBuildFromCellListParallel. >>>>>>>>> >>>>>>>>> I would like this interger to be distribruted by or the same way >>>>>>>>> DMPlexDistribute distribute the mesh. >>>>>>>>> >>>>>>>>> Is it possible to do this? >>>>>>>>> >>>>>>>> >>>>>>>> I think we already have support for what you want. If you call >>>>>>>> >>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >>>>>>>> >>>>>>>> before DMPlexDistribute(), it will compute a PetscSF encoding the >>>>>>>> global to natural map. You >>>>>>>> can get it with >>>>>>>> >>>>>>>> >>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >>>>>>>> >>>>>>>> and use it with >>>>>>>> >>>>>>>> >>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >>>>>>>> >>>>>>>> Is this sufficient? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> On 2021-07-14 1:18 p.m., Eric Chamberland wrote: >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I want to use DMPlexDistribute from PETSc for computing >>>>>>>>> overlapping >>>>>>>>> > and play with the different partitioners supported. >>>>>>>>> > >>>>>>>>> > However, after calling DMPlexDistribute, I noticed the elements >>>>>>>>> are >>>>>>>>> > renumbered and then the original number is lost. >>>>>>>>> > >>>>>>>>> > What would be the best way to keep track of the element >>>>>>>>> renumbering? >>>>>>>>> > >>>>>>>>> > a) Adding an optional parameter to let the user retrieve a >>>>>>>>> vector or >>>>>>>>> > "IS" giving the old number? >>>>>>>>> > >>>>>>>>> > b) Adding a DMLabel (seems a wrong good solution) >>>>>>>>> > >>>>>>>>> > c) Other idea? >>>>>>>>> > >>>>>>>>> > Of course, I don't want to loose performances with the need of >>>>>>>>> this >>>>>>>>> > "mapping"... >>>>>>>>> > >>>>>>>>> > Thanks, >>>>>>>>> > >>>>>>>>> > Eric >>>>>>>>> > >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Universit? Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Universit? Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Eric Chamberland, ing., M. Ing >>>>>>> Professionnel de recherche >>>>>>> GIREF/Universit? Laval >>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Universit? Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Universit? Laval >>>>> (418) 656-2131 poste 41 22 42 >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbnbhlbilhmjdpfg.png Type: image/png Size: 42972 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eejjfmbjimlkboec.png Type: image/png Size: 87901 bytes Desc: not available URL: From nicolas.tardieu at edf.fr Wed Nov 3 07:06:51 2021 From: nicolas.tardieu at edf.fr (TARDIEU Nicolas) Date: Wed, 3 Nov 2021 12:06:51 +0000 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: Dear Matt, Thank you for the additional information. I do agree that I will have to write code to describe the interpolation I need for a multi-model problem. Let me try to illustrate my point on the simple mesh I have attached to this email. It is done with Gmsh but other generators I use can do the same. The mesh is composed of : - a 3D part, volume_3d, where continuum mechanics equations will be assigned (displacement dof) - a 2D part, neumann_2d, which is a face of volume_3d and that will be used to integrate a Neumann condition assigned to volume_3d (it shares the displacement dof of volume_3d) - a 2D part, shell_2d, where shell equations will be assigned (displacement+rotation dof) - a 1D part, beam_1d, where beam equations will be assigned (displacement+rotation dof) I can imagine that it is impossible, only from the knowledge of the mesh, to guess the interpolation. But I would be happy to try to write a small DMPlex test to try to describe such a model with a little help from you or another PETSc practitioner. Regards, Nicolas ________________________________ De : knepley at gmail.com Envoy? : mercredi 3 novembre 2021 12:14 ? : TARDIEU Nicolas Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] How to construct DMPlex of cells with different topological dimension? On Wed, Nov 3, 2021 at 4:20 AM TARDIEU Nicolas via petsc-users > wrote: Dear all, As a practitioner of structural mechanics, I modestly wish to support Yuan's argument. Model mixing, and consequently finite element dimension mixing, is (perhaps unfortunately) an inescapable practice of this engineering field. This has consequences not only on the nature of the meshes we use but also on the quasi-systematic recourse to linear relations (also named multipoints constraints, MPC) between the degrees of freedom of the different models (for example, linking a shell to a 3D part requires to write kinematic connections between displacements and rotations). DMPlex handling such meshes would be a really great contribution for the structural mechanics community. Nicolas, >From my point of view, this is supported, so it would be good to understand what you do not think is supported: 1) Multiple models This is supported in the sense that you can use any data layout, and assemble any system you want into it. You can use beams along edges and shells on faces if you want. I have examples of different equations in different pieces of the domain. 2) Mesh with pieces hanging off This is supported in the sense that you can input such a mesh. You have to describe it completely, but it can be input and iterated over. 3) Mesh Interpolation If you only want to input vertices _and_ you want to input different dimensional objects, you will have to write code. There is no way to guess what you mean from a collection of vertices. You could a) Directly call GetRawFaces() to do the interpolation per cell before input (seems easier) b) Rewrite the interpolation loop to check whether cell types have been set and just call the appropriate interpolation if they have I guess you are arguing that I should add the code for 3). Most people I have interacted with use a mesh generator to make the meshes, so I concentrated on importing those formats. Can meshes like these be made with those generators? Thanks, Matt Best regards, Nicolas Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: multimodel.geo Type: application/octet-stream Size: 767 bytes Desc: multimodel.geo URL: From knepley at gmail.com Wed Nov 3 07:12:36 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 3 Nov 2021 08:12:36 -0400 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: On Wed, Nov 3, 2021 at 8:06 AM TARDIEU Nicolas wrote: > Dear Matt, > > Thank you for the additional information. > I do agree that I will have to write code to describe the interpolation I > need for a multi-model problem. > Let me try to illustrate my point on the simple mesh I have attached to > this email. It is done with Gmsh but other generators I use can do the same. > The mesh is composed of : > > - a 3D part, volume_3d, where continuum mechanics equations will be > assigned (displacement dof) > - a 2D part, neumann_2d, which is a face of volume_3d and that will be > used to integrate a Neumann condition assigned to volume_3d (it shares the > displacement dof of volume_3d) > - a 2D part, shell_2d, where shell equations will be assigned > (displacement+rotation dof) > - a 1D part, beam_1d, where beam equations will be assigned > (displacement+rotation dof) > > I can imagine that it is impossible, only from the knowledge of the mesh, > to guess the interpolation. But I would be happy to try to write a small > DMPlex test to try to describe such a model with a little help from you or > another PETSc practitioner. > That is a good idea. I think we can use the pieces we have to build what you want automatically. Cn you send me the *.msh for this. I do not even have GMsh running on my machine, so I cannot process the .geo :) When we read in GMah files, we mark the cell type. However, I think we automatically try to find lower dimensional pieces in the existing mesh (since that is what people usually want, like your neumann section), so we would need a way to indicate that we are specifying lower dimensional pieces that are not part of a volumetric piece. Thanks, Matt > Regards, > Nicolas > ------------------------------ > *De :* knepley at gmail.com > *Envoy? :* mercredi 3 novembre 2021 12:14 > *? :* TARDIEU Nicolas > *Cc :* petsc-users at mcs.anl.gov > *Objet :* Re: [petsc-users] How to construct DMPlex of cells with > different topological dimension? > > On Wed, Nov 3, 2021 at 4:20 AM TARDIEU Nicolas via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Dear all, > > As a practitioner of structural mechanics, I modestly wish to support > Yuan's argument. Model mixing, and consequently finite element dimension > mixing, is (perhaps unfortunately) an inescapable practice of this > engineering field. This has consequences not only on the nature of the > meshes we use but also on the quasi-systematic recourse to linear relations > (also named multipoints constraints, MPC) between the degrees of freedom of > the different models (for example, linking a shell to a 3D part requires to > write kinematic connections between displacements and rotations). > DMPlex handling such meshes would be a really great contribution for the > structural mechanics community. > > > Nicolas, > > From my point of view, this is supported, so it would be good to > understand what you do not think is supported: > > 1) Multiple models > > This is supported in the sense that you can use any data layout, and > assemble any system you want into it. You can use > beams along edges and shells on faces if you want. I have examples of > different equations in different pieces of the domain. > > 2) Mesh with pieces hanging off > > This is supported in the sense that you can input such a mesh. You > have to describe it completely, but it can be input and iterated over. > > 3) Mesh Interpolation > > If you only want to input vertices _and_ you want to input different > dimensional objects, you will have to write code. There is no way to guess > what you mean from a collection of vertices. You could > > a) Directly call GetRawFaces() to do the interpolation per cell > before input (seems easier) > > b) Rewrite the interpolation loop to check whether cell types have > been set and just call the appropriate interpolation if they have > > I guess you are arguing that I should add the code for 3). Most people I > have interacted with use a mesh generator to make the meshes, so > I concentrated on importing those formats. Can meshes like these be made > with those generators? > > Thanks, > > Matt > > > Best regards, > Nicolas > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de > le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont > ?tablis ? l'intention exclusive des destinataires et les informations qui y > figurent sont strictement confidentielles. Toute utilisation de ce Message > non conforme ? sa destination, toute diffusion ou toute publication totale > ou partielle, est interdite sauf autorisation expresse. > > Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de > le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou > partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de > votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace > sur quelque support que ce soit. Nous vous remercions ?galement d'en > avertir imm?diatement l'exp?diteur par retour du message. > > Il est impossible de garantir que les communications par messagerie > ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute > erreur ou virus. > ____________________________________________________ > > This message and any attachments (the 'Message') are intended solely for > the addressees. The information contained in this Message is confidential. > Any use of information contained in this Message not in accord with its > purpose, any dissemination or disclosure, either whole or partial, is > prohibited except formal approval. > > If you are not the addressee, you may not copy, forward, disclose or use > any part of it. If you have received this message in error, please delete > it and all copies from your system and notify the sender immediately by > return message. > > E-mail communication cannot be guaranteed to be timely secure, error or > virus-free. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.tardieu at edf.fr Wed Nov 3 08:25:04 2021 From: nicolas.tardieu at edf.fr (TARDIEU Nicolas) Date: Wed, 3 Nov 2021 13:25:04 +0000 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: Here are the msh files. In the first one, each different part is meshed with a single element. The others are subdivisions of the original one. Let me know if you need any extra stuff. Regards, Nicolas ________________________________ De : knepley at gmail.com Envoy? : mercredi 3 novembre 2021 13:12 ? : TARDIEU Nicolas Cc : petsc-users at mcs.anl.gov Objet : Re: [petsc-users] How to construct DMPlex of cells with different topological dimension? On Wed, Nov 3, 2021 at 8:06 AM TARDIEU Nicolas > wrote: Dear Matt, Thank you for the additional information. I do agree that I will have to write code to describe the interpolation I need for a multi-model problem. Let me try to illustrate my point on the simple mesh I have attached to this email. It is done with Gmsh but other generators I use can do the same. The mesh is composed of : - a 3D part, volume_3d, where continuum mechanics equations will be assigned (displacement dof) - a 2D part, neumann_2d, which is a face of volume_3d and that will be used to integrate a Neumann condition assigned to volume_3d (it shares the displacement dof of volume_3d) - a 2D part, shell_2d, where shell equations will be assigned (displacement+rotation dof) - a 1D part, beam_1d, where beam equations will be assigned (displacement+rotation dof) I can imagine that it is impossible, only from the knowledge of the mesh, to guess the interpolation. But I would be happy to try to write a small DMPlex test to try to describe such a model with a little help from you or another PETSc practitioner. That is a good idea. I think we can use the pieces we have to build what you want automatically. Cn you send me the *.msh for this. I do not even have GMsh running on my machine, so I cannot process the .geo :) When we read in GMah files, we mark the cell type. However, I think we automatically try to find lower dimensional pieces in the existing mesh (since that is what people usually want, like your neumann section), so we would need a way to indicate that we are specifying lower dimensional pieces that are not part of a volumetric piece. Thanks, Matt Regards, Nicolas ________________________________ De : knepley at gmail.com > Envoy? : mercredi 3 novembre 2021 12:14 ? : TARDIEU Nicolas > Cc : petsc-users at mcs.anl.gov > Objet : Re: [petsc-users] How to construct DMPlex of cells with different topological dimension? On Wed, Nov 3, 2021 at 4:20 AM TARDIEU Nicolas via petsc-users > wrote: Dear all, As a practitioner of structural mechanics, I modestly wish to support Yuan's argument. Model mixing, and consequently finite element dimension mixing, is (perhaps unfortunately) an inescapable practice of this engineering field. This has consequences not only on the nature of the meshes we use but also on the quasi-systematic recourse to linear relations (also named multipoints constraints, MPC) between the degrees of freedom of the different models (for example, linking a shell to a 3D part requires to write kinematic connections between displacements and rotations). DMPlex handling such meshes would be a really great contribution for the structural mechanics community. Nicolas, >From my point of view, this is supported, so it would be good to understand what you do not think is supported: 1) Multiple models This is supported in the sense that you can use any data layout, and assemble any system you want into it. You can use beams along edges and shells on faces if you want. I have examples of different equations in different pieces of the domain. 2) Mesh with pieces hanging off This is supported in the sense that you can input such a mesh. You have to describe it completely, but it can be input and iterated over. 3) Mesh Interpolation If you only want to input vertices _and_ you want to input different dimensional objects, you will have to write code. There is no way to guess what you mean from a collection of vertices. You could a) Directly call GetRawFaces() to do the interpolation per cell before input (seems easier) b) Rewrite the interpolation loop to check whether cell types have been set and just call the appropriate interpolation if they have I guess you are arguing that I should add the code for 3). Most people I have interacted with use a mesh generator to make the meshes, so I concentrated on importing those formats. Can meshes like these be made with those generators? Thanks, Matt Best regards, Nicolas Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ Ce message et toutes les pi?ces jointes (ci-apr?s le 'Message') sont ?tablis ? l'intention exclusive des destinataires et les informations qui y figurent sont strictement confidentielles. Toute utilisation de ce Message non conforme ? sa destination, toute diffusion ou toute publication totale ou partielle, est interdite sauf autorisation expresse. Si vous n'?tes pas le destinataire de ce Message, il vous est interdit de le copier, de le faire suivre, de le divulguer ou d'en utiliser tout ou partie. Si vous avez re?u ce Message par erreur, merci de le supprimer de votre syst?me, ainsi que toutes ses copies, et de n'en garder aucune trace sur quelque support que ce soit. Nous vous remercions ?galement d'en avertir imm?diatement l'exp?diteur par retour du message. Il est impossible de garantir que les communications par messagerie ?lectronique arrivent en temps utile, sont s?curis?es ou d?nu?es de toute erreur ou virus. ____________________________________________________ This message and any attachments (the 'Message') are intended solely for the addressees. The information contained in this Message is confidential. Any use of information contained in this Message not in accord with its purpose, any dissemination or disclosure, either whole or partial, is prohibited except formal approval. If you are not the addressee, you may not copy, forward, disclose or use any part of it. If you have received this message in error, please delete it and all copies from your system and notify the sender immediately by return message. E-mail communication cannot be guaranteed to be timely secure, error or virus-free. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: multimodelX4.msh Type: application/octet-stream Size: 7759 bytes Desc: multimodelX4.msh URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: multimodelX2.msh Type: application/octet-stream Size: 2530 bytes Desc: multimodelX2.msh URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: multimodel.msh Type: application/octet-stream Size: 1580 bytes Desc: multimodel.msh URL: From ghosh5 at llnl.gov Wed Nov 3 14:28:42 2021 From: ghosh5 at llnl.gov (Ghosh, Debojyoti) Date: Wed, 3 Nov 2021 19:28:42 +0000 Subject: [petsc-users] PETSc with SuperLU_dist Message-ID: Hi, I am trying to configure PETSc with SuperLU_dist, but the configure step exits with an error while compiling SuperLU_dist. I am doing this on LLNL's Quartz. The configure command is: ./configure --with-batch --with-cc=mpicc --with-fc=mpif90 -with-cxx=mpicxx COPTFLAGS="-O2" FOPTFLAGS="-O2" CXXOPTFLAGS="-O2" --with-shared-libraries --with-debugging=0 --download-cmake --download-hypre --download-superlu --download-superlu_dist --download-parmetis --download-metis I have appended the version information for my compilers below. Also attached is the configure.log. Can you please let me know what's going wrong? Thanks, Debo -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 1480034 bytes Desc: configure.log URL: From balay at mcs.anl.gov Wed Nov 3 14:38:11 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 3 Nov 2021 14:38:11 -0500 (CDT) Subject: [petsc-users] PETSc with SuperLU_dist In-Reply-To: References: Message-ID: <24e31774-3754-c232-546c-913684e943c@mcs.anl.gov> hi Debo, >>> In file included from /g/g92/ghosh5/Codes/petsc/arch-opt-quartz/include/superlu_defs.h:113:0, from /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c:14: /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c: In function 'f_create_gridinfo3d_handle': /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_dwrap.c: In function 'f_dDestroy_LU_SOLVE_struct_3d': /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c:57:43: error: 'gridinfo3d_t' undeclared (first use in this function) *handle = (fptr) SUPERLU_MALLOC(sizeof(gridinfo3d_t)); <<<< Did you have a prior build of superlu or superlu_dist in this location? Can you try a new build [say with a different PETSC_ARCH - and see if the issue persists? Satish On Wed, 3 Nov 2021, Ghosh, Debojyoti via petsc-users wrote: > Hi, > > I am trying to configure PETSc with SuperLU_dist, but the configure step exits with an error while compiling SuperLU_dist. I am doing this on LLNL's Quartz. > > The configure command is: > ./configure --with-batch --with-cc=mpicc --with-fc=mpif90 -with-cxx=mpicxx COPTFLAGS="-O2" FOPTFLAGS="-O2" CXXOPTFLAGS="-O2" --with-shared-libraries --with-debugging=0 --download-cmake --download-hypre --download-superlu --download-superlu_dist --download-parmetis --download-metis > > I have appended the version information for my compilers below. Also attached is the configure.log. Can you please let me know what's going wrong? > > Thanks, > Debo > From ghosh5 at llnl.gov Wed Nov 3 14:58:11 2021 From: ghosh5 at llnl.gov (Ghosh, Debojyoti) Date: Wed, 3 Nov 2021 19:58:11 +0000 Subject: [petsc-users] PETSc with SuperLU_dist In-Reply-To: <24e31774-3754-c232-546c-913684e943c@mcs.anl.gov> References: <24e31774-3754-c232-546c-913684e943c@mcs.anl.gov> Message-ID: <6C2252AC-CCCD-4CC9-87B9-2E6471530571@llnl.gov> Thanks, Satish! I was able to configure successfully with a different PETSC_ARCH. And yes, I looked at the timestamps in my "/usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist" directory, and some files are from a few months back. I'll now delete my original PETSC_ARCH and retry; I assume it should work. ?On 11/3/21, 12:38 PM, "Satish Balay" wrote: hi Debo, >>> In file included from /g/g92/ghosh5/Codes/petsc/arch-opt-quartz/include/superlu_defs.h:113:0, from /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c:14: /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c: In function 'f_create_gridinfo3d_handle': /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_dwrap.c: In function 'f_dDestroy_LU_SOLVE_struct_3d': /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c:57:43: error: 'gridinfo3d_t' undeclared (first use in this function) *handle = (fptr) SUPERLU_MALLOC(sizeof(gridinfo3d_t)); <<<< Did you have a prior build of superlu or superlu_dist in this location? Can you try a new build [say with a different PETSC_ARCH - and see if the issue persists? Satish On Wed, 3 Nov 2021, Ghosh, Debojyoti via petsc-users wrote: > Hi, > > I am trying to configure PETSc with SuperLU_dist, but the configure step exits with an error while compiling SuperLU_dist. I am doing this on LLNL's Quartz. > > The configure command is: > ./configure --with-batch --with-cc=mpicc --with-fc=mpif90 -with-cxx=mpicxx COPTFLAGS="-O2" FOPTFLAGS="-O2" CXXOPTFLAGS="-O2" --with-shared-libraries --with-debugging=0 --download-cmake --download-hypre --download-superlu --download-superlu_dist --download-parmetis --download-metis > > I have appended the version information for my compilers below. Also attached is the configure.log. Can you please let me know what's going wrong? > > Thanks, > Debo > From balay at mcs.anl.gov Wed Nov 3 15:05:43 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 3 Nov 2021 15:05:43 -0500 (CDT) Subject: [petsc-users] PETSc with SuperLU_dist In-Reply-To: <6C2252AC-CCCD-4CC9-87B9-2E6471530571@llnl.gov> References: <24e31774-3754-c232-546c-913684e943c@mcs.anl.gov> <6C2252AC-CCCD-4CC9-87B9-2E6471530571@llnl.gov> Message-ID: Yes deleting current build-files/PETSC_ARCH should also work! Satish On Wed, 3 Nov 2021, Ghosh, Debojyoti via petsc-users wrote: > Thanks, Satish! I was able to configure successfully with a different PETSC_ARCH. > > And yes, I looked at the timestamps in my "/usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist" directory, and some files are from a few months back. I'll now delete my original PETSC_ARCH and retry; I assume it should work. > > ?On 11/3/21, 12:38 PM, "Satish Balay" wrote: > > hi Debo, > > >>> > In file included from /g/g92/ghosh5/Codes/petsc/arch-opt-quartz/include/superlu_defs.h:113:0, > from /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c:14: > /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c: In function 'f_create_gridinfo3d_handle': > /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_dwrap.c: In function 'f_dDestroy_LU_SOLVE_struct_3d': > /usr/WS2/ghosh5/Codes/petsc/arch-opt-quartz/externalpackages/git.superlu_dist/FORTRAN/superlu_c2f_wrap.c:57:43: error: 'gridinfo3d_t' undeclared (first use in this function) > *handle = (fptr) SUPERLU_MALLOC(sizeof(gridinfo3d_t)); > <<<< > > Did you have a prior build of superlu or superlu_dist in this location? > > Can you try a new build [say with a different PETSC_ARCH - and see if the issue persists? > > Satish > > > On Wed, 3 Nov 2021, Ghosh, Debojyoti via petsc-users wrote: > > > Hi, > > > > I am trying to configure PETSc with SuperLU_dist, but the configure step exits with an error while compiling SuperLU_dist. I am doing this on LLNL's Quartz. > > > > The configure command is: > > ./configure --with-batch --with-cc=mpicc --with-fc=mpif90 -with-cxx=mpicxx COPTFLAGS="-O2" FOPTFLAGS="-O2" CXXOPTFLAGS="-O2" --with-shared-libraries --with-debugging=0 --download-cmake --download-hypre --download-superlu --download-superlu_dist --download-parmetis --download-metis > > > > I have appended the version information for my compilers below. Also attached is the configure.log. Can you please let me know what's going wrong? > > > > Thanks, > > Debo > > > > > From yuanxi at advancesoft.jp Wed Nov 3 20:19:15 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Thu, 4 Nov 2021 10:19:15 +0900 Subject: [petsc-users] How to construct DMPlex of cells with different topological dimension? In-Reply-To: References: Message-ID: I think the above Jed's reply declared what I am worried about. That is: if I don't use faking 3D elements, automatic topological interpolation is not available. I also notice your discussion with Nicolas. You suggest users could do it themselves by using GetRawFaces etc. I think maybe I can try to do it another time. Thank you very much Yuan 2021?11?3?(?) 20:19 Matthew Knepley : > On Tue, Nov 2, 2021 at 10:49 PM ?? wrote: > >> Well! All objects in this world could definitely be modeled as 3D >> volumes. But considering costs of meshing and following calculations, in >> most cases it is neither practical nor necessary. And in some specific >> cases, e.g., to model cell membranes, 2D membrane elements may behave >> better than 3D volume ones. It is therefore, in many cases, modeling should >> be much simplified and in some cases they could be modeled as 2D or 1D ones. >> > > I have _never_ suggested using 3D elements, as you can see from my reply > to Nicolas. I said you can use _exactly_ the element you want. I am not > sure how to be more clear. > > Matt > > >> Following paper is obtained after just google "parachute, FSI" >> *A parallel 3D computational method for fluid-structure interactions in >> parachute systems >> * >> >> You can see they use cable (1D) and membrane (2D) elements to model >> parachutes. Those public software, such as ABAQUS, ANSYS, NASTRAN, etc all >> allow the mixed usage of topologically different elements. They tell the >> same story. It is the reality and I don't think such reality would change >> in the near future. >> >> Best regards >> >> Yuan >> >> 2021?11?2?(?) 18:22 Matthew Knepley : >> >>> On Mon, Nov 1, 2021 at 11:20 PM ?? wrote: >>> >>>> Dear Matthew, >>>> >>>> I built a test problem using the strategy you suggested. It works! It >>>> is enough for me right now. Thank you very much. >>>> >>>> ! 9----------8----------13 >>>> ! /| /| /| >>>> ! / | / | / | >>>> ! / | / | / | >>>> ! 6---------7---------12 | >>>> ! | | | | | | >>>> ! | | | | | | >>>> ! | | | | | | >>>> ! | | | | | | >>>> ! | 5------|---4-------|--11--------17------16 >>>> ! | / | / | / / / >>>> ! | / | / | / / / >>>> ! |/ |/ |/ / / >>>> ! 2-- ------3---------10--------14-------15 >>>> >>>> coneSize = (/ 8,8,8,8, 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0 /) >>>> cones = (/ 2,5,4,3,6,7,8,9, 3,4,11,10,7,12,13,8, & >>>> 10,14,17,11, 10,14,17,11, 14,15,16,17, 14,15,16,17 /) >>>> >>>> There is still a problem left, however, that is how to build a 3D cell >>>> from a 2 vertex segment. I think I could construct a virtual tetrahedron. >>>> >>>> Finally, It can't be denied that this approach needs additional effort >>>> in mesh construction (and It is something strange to construct a 3D object >>>> from a one dimensional segment). In fact, structures with different >>>> topological dimensions are not that rare. Parachute, for example, may be >>>> composed with two dimensional parafoil and one dimensional cord. FRP(Fiber >>>> reinforced plastics) may be modelled by one dimensional reinforcement and >>>> three dimensional plastics. It is therefore, I am wondering, if PETSc would >>>> take into consider of such cases by, for example >>>> >>>> - Enable a face, DM_POLYTOPE_SEGMENT2D, defined by one edge. And >>>> - Eable cells, such >>>> as DM_POLYTOPE_QUADRILATERAL3D, DM_POLYTOPE_TRIANGLE3D and >>>> DM_POLYTOPE_SEGMENT3D, with one face. >>>> >>> >>> I think I am still not being completely clear. These types of cells >>> definitely exist. However, let's take the case of the parachute. The one >>> dimensional cords are part of >>> the two dimensional patches, which are part of a three dimensional >>> volume. This is how we have modeled it. You assemble different equations on >>> different pieces, but >>> that does not affect the mesh. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Sorry for a newbie in PETSc to provide such a suggestion. But if it >>>> could be accomplished, it would help mech structural engineering, civil >>>> engineering, and solid mechanics applicationers. >>>> >>>> Best regards, >>>> >>>> Yuan >>>> >>>> >>>> >>>> 2021?11?1?(?) 18:32 Matthew Knepley : >>>> >>>>> On Sun, Oct 31, 2021 at 12:21 PM ?? wrote: >>>>> >>>>>> Dear Matt >>>>>> >>>>>> Thank you for your detailed explanation. >>>>>> >>>>>> First, I would like to answer your question about my application >>>>>> where low dimensional features are not embedded in the larger volume. It is >>>>>> quite general in structural engineering. For example, buildings are >>>>>> generally modelled as shells and beams, which are two and one dimension >>>>>> respectively. While building foundation is modelled by solid >>>>>> elements, which is three dimension, at the same time. >>>>>> >>>>> >>>>> I think I see what you want now. Let me make a suggestion (along the >>>>> lines of what Mark said), and attempt to justify it by answering some >>>>> questions. >>>>> >>>>> Suggestion: I think you should consider using a volumetric mesh for >>>>> your problem >>>>> >>>>> Q1: Can I get the same algebraic system this way? >>>>> >>>>> Yes. You would only assign unknowns to faces and edges of the mesh >>>>> where you have shell and beam elements. >>>>> >>>>> Q2: What are the advantages? >>>>> >>>>> You would get topological connectivity of all parts of the structure, >>>>> automatic coupling of the different formulations, >>>>> and you could separately solve the different structures using block >>>>> preconditioners, while still forming a unified >>>>> residual so that you can guarantee a consistent solution. >>>>> >>>>> Q3: Wouldn't this be expensive? >>>>> >>>>> No. For the shells and beams, you would still need the vertices, >>>>> edges, and faces. First, by the Euler relation, these would >>>>> outnumbers the additional cells. Second, since no unknowns would be >>>>> associated with the cells, only additional memory in >>>>> the mesh would be used, not the system solves. Memory and time are >>>>> dominated by the solve, so this should be in the noise. >>>>> >>>>> What do you think? >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Secondly, It is regrettably to find that DMComposite is not available >>>>>> for me, because all my solid, shell, and beam elements are connected each >>>>>> other. >>>>>> >>>>>> At last, I have build a simple program to see if DMPlexSetCellType() >>>>>> works for me, following the suggestion of functions in PETSc like >>>>>> DMPlexCreateCGNS. But it failed when it tried to do DMPlexInterpolate >>>>>> ! 9----------8---------13 >>>>>> ! /| /| /| >>>>>> ! / | / | / | >>>>>> ! / | / | / | >>>>>> ! 6---------7---------12 | >>>>>> ! | | | | | | >>>>>> ! | | | | | | >>>>>> ! | | | | | | >>>>>> ! | | | | | | >>>>>> ! | 5------|---4-------|-11--------17--------16 >>>>>> ! | / | / | / / / >>>>>> ! | / | / | / / / >>>>>> ! |/ |/ |/ / / >>>>>> ! 2---------3---------10--------14-------15 >>>>>> >>>>>> The calculation result are follows. It seems that the cell type are >>>>>> set correctly but depth is still 2. >>>>>> -------------------------------------------------------------------- >>>>>> DM Object: TestMesh 2 MPI processes >>>>>> type: plex >>>>>> TestMesh in 3 dimensions: >>>>>> 0-cells: 16 0 >>>>>> 3-cells: 20 (18) 0 >>>>>> Labels: >>>>>> celltype: 3 strata with value/size (7 (2), 4 (2), 0 (16)) >>>>>> depth: 2 strata with value/size (0 (16), 1 (20)) >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>> [0]PETSC ERROR: Array was not checked out >>>>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble >>>>>> shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>> v3.16.0-351-g743e004674 GIT Date: 2021-10-29 09:32:23 -0500 >>>>>> [0]PETSC ERROR: ./ex3f90 on a arch-linux-c-debug named >>>>>> DESKTOP-9ITFSBM by hillyuan Mon Nov 1 00:26:39 2021 >>>>>> [0]PETSC ERROR: Configure options --with-cc=mpiicc --with-cxx=mpiicpc >>>>>> --with-fc=mpiifort --with-fortran-bindings=1 >>>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.3.0 >>>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.3.0 >>>>>> [0]PETSC ERROR: #1 DMRestoreWorkArray() at >>>>>> /home/hillyuan/programs/petsc/src/dm/interface/dm.c:1580 >>>>>> [0]PETSC ERROR: #2 DMPlexRestoreRawFaces_Internal() at >>>>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:323 >>>>>> [0]PETSC ERROR: #3 DMPlexInterpolateFaces_Internal() at >>>>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:375 >>>>>> [0]PETSC ERROR: #4 DMPlexInterpolate() at >>>>>> /home/hillyuan/programs/petsc/src/dm/impls/plex/plexinterpolate.c:1340 >>>>>> >>>>>> ----------------------------------------------------------------------------------------- >>>>>> I attached my test program in this mail. It is much appreciated that >>>>>> you could provide any suggestion. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Yuan >>>>>> >>>>>> >>>>>> 2021?10?31?(?) 21:16 Matthew Knepley : >>>>>> >>>>>>> On Thu, Oct 28, 2021 at 10:48 PM ?? wrote: >>>>>>> >>>>>>>> Dear Matt, >>>>>>>> >>>>>>>> My mesh is something like the following figure, which is >>>>>>>> composed of three elements : one hexahedron(solid element), one >>>>>>>> quadrilateral (shell element), and one line (beam element). I found the >>>>>>>> function "TestEmptyStrata" in file \dm\impls\plex\tests\ex11.c >>>>>>>> would be a good example to read in such a kind of mesh by using >>>>>>>> DMPlexSetCone. But a problem is that you should declare all faces and edges >>>>>>>> of hexahedron element, all edges in quadrilateral element by >>>>>>>> DMPlexSetCone, otherwise PETsc could not do >>>>>>>> topological interpolation afterwards. Am I right here? >>>>>>>> As general in FEM mesh, my mesh does not contain any information >>>>>>>> about faces or edges of solid elements. That's why I consider >>>>>>>> using DMCOMPOSITE. That is >>>>>>>> >>>>>>>> - Put hexahedron, quadrilateral, and line elements into >>>>>>>> different DM structures. >>>>>>>> - do topological interpolation in those DMs separately. >>>>>>>> - composite them. >>>>>>>> >>>>>>>> Is there anything wrong in my above consideration? Any >>>>>>>> suggestions? >>>>>>>> >>>>>>>> ------------ >>>>>>>> /| /| >>>>>>>> / | / | cell 0: Hex >>>>>>>> / | / | >>>>>>>> ------------/ | >>>>>>>> | | | | >>>>>>>> | | | | cell 1: Quad >>>>>>>> | --------|---|------------ >>>>>>>> | / | / / >>>>>>>> | / | / / >>>>>>>> |/ |/ / >>>>>>>> ------------------------------------------- >>>>>>>> cell 2: line >>>>>>>> >>>>>>>> Much thanks for your help. >>>>>>>> >>>>>>> >>>>>>> If you are solving something where everything is embedded in a >>>>>>> volumetric mesh, then there is no problem. However, if you really have >>>>>>> the mesh above, where lower dimensional pieces are sticking out of >>>>>>> the mesh, then Plex can represent the mesh, but automatic interpolation >>>>>>> (creation of edges and faces) will not work. Why is this? We use >>>>>>> depth in the DAG as a proxy for cell dimension, but this will no longer work >>>>>>> if faces are not part of a volume. >>>>>>> >>>>>>> Will DMCOMPOSITE do what you want? It depends. It will be able to >>>>>>> lay out a vector, but it will not know about any topological connectivity >>>>>>> between the meshes and will not preallocate a Jacobian with any >>>>>>> interaction. If the meshes are truly separate, this is fine. If not, it is >>>>>>> not that >>>>>>> useful. >>>>>>> >>>>>>> Could you modify the existing code to support this? Yes, it would >>>>>>> not be terribly difficult. When you load the mesh, you must know what kind >>>>>>> of cell you are loading. You could explicitly set this using >>>>>>> DMPlexSetCellType(). Then, instead of taking a certain height stratum of >>>>>>> the DAG >>>>>>> to loop over, you would instead use all cells marked with a certain >>>>>>> cell type. The rest of the interpolation code should work fine. >>>>>>> >>>>>>> What kind of physics do you have where low dimensional features are >>>>>>> not embedded in the larger volume? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> Yuan >>>>>>>> >>>>>>>> 2021?10?28?(?) 22:05 Matthew Knepley : >>>>>>>> >>>>>>>>> On Thu, Oct 28, 2021 at 4:59 AM ?? wrote: >>>>>>>>> >>>>>>>>>> Dear Matt, >>>>>>>>>> >>>>>>>>>> Thank you for your quick response. >>>>>>>>>> >>>>>>>>>> I think what you mean is to build DAG from my mesh at first and >>>>>>>>>> then call DMPlexCreateFromDAG >>>>>>>>>> () >>>>>>>>>> to construct DMPlex. >>>>>>>>>> >>>>>>>>> >>>>>>>>> No, I do not mean that. >>>>>>>>> >>>>>>>>> >>>>>>>>>> A new problem is, as I know, the function DMPlexInterpolate would >>>>>>>>>> generate points with different depth. What's the difference between those >>>>>>>>>> faces and segment elements generated by DMPlexInterpolate with that >>>>>>>>>> defined by the original mesh, or should we not use DMPlexInterpolate in >>>>>>>>>> such a case? >>>>>>>>>> >>>>>>>>>> On the other hand, can DMComposite be used in this case by >>>>>>>>>> defining DMPlex with different topological dimensions at first and then >>>>>>>>>> composite them? >>>>>>>>>> >>>>>>>>> >>>>>>>>> You do not need that. I am obviously not understanding your >>>>>>>>> question. My short answer is that Plex _already_ handles cells of different >>>>>>>>> dimension automatically without anything extra. >>>>>>>>> >>>>>>>>> Maybe it would help if you defined a specific problem you have. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks in advance. >>>>>>>>>> >>>>>>>>>> Yuan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2021?10?27?(?) 19:27 Matthew Knepley : >>>>>>>>>> >>>>>>>>>>> On Wed, Oct 27, 2021 at 4:50 AM ?? >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I am trying to parallelize my serial FEM program using PETSc. >>>>>>>>>>>> This program calculates structure deformation by using various types of >>>>>>>>>>>> elements such as solid, shell, beam, and truss. At the very beginning, I >>>>>>>>>>>> found it was hard for me to put such kinds of elements into DMPlex. Because >>>>>>>>>>>> solid elements are topologically three dimensional, shell element two, and >>>>>>>>>>>> beam or truss are topologically one-dimensional elements. After reading >>>>>>>>>>>> chapter 2.10: "DMPlex: Unstructured Grids in PETSc" of users manual >>>>>>>>>>>> carefully, I found the provided functions, such as DMPlexSetCone, cannot >>>>>>>>>>>> declare those topological differences. >>>>>>>>>>>> >>>>>>>>>>>> My question is : Is it possible and how to define all those >>>>>>>>>>>> topologically different elements into a DMPlex struct? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes. The idea is to program in a dimension-independent way, so >>>>>>>>>>> that the code can handle cells of any dimension. >>>>>>>>>>> What you probably want is the "depth" in the DAG representation, >>>>>>>>>>> which you can think of as the dimension of a cell. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetPointDepth.html#DMPlexGetPointDepth >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thanks in advance! >>>>>>>>>>>> >>>>>>>>>>>> Best regards, >>>>>>>>>>>> >>>>>>>>>>>> Yuan. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Nov 3 22:32:29 2021 From: jed at jedbrown.org (Jed Brown) Date: Wed, 03 Nov 2021 21:32:29 -0600 Subject: [petsc-users] Tutorials test case cannot run in parallel In-Reply-To: References: Message-ID: <87tugsmw8i.fsf@jedbrown.org> Mark Adams writes: > Do we have a policy on this? I know some tests say they are serial. > Maybe just say that tests are only supported for the parameters in the test. If a test is serial, it should check the size of PETSC_COMM_WORLD and error if it's bigger than 1. It doesn't help to have a structured way to tell the test suite that it can't run in parallel -- the test suite runs only what we tell it to and users run the examples outside the test suite so it needs to error. Lots of examples do this. From lulu.liu at kaust.edu.sa Sat Nov 6 06:19:59 2021 From: lulu.liu at kaust.edu.sa (Lulu Liu) Date: Sat, 6 Nov 2021 19:19:59 +0800 Subject: [petsc-users] The exact Jacobian for ASPIN Message-ID: Hi, For ASPIN, the local problem F_{j}(x-T_{j})=0 (j=1,2,...N) is solved. The exact Jacobian of the preconditioned Jacobian is J_exact=\sum_{j=1}^{N}J_{I}(x-T_{i})J(x-T_{j}). It seems that PETSc uses the approximate Jacobian, J_exact=\sum_{j=1}^{N}J_{j}(x-T_{i})J(x-\sum_{j=1}^{N}T_{j}). I want to implement RASPEN, which requires the exact Jacobian of ASPIN. Is there any easy way to compute J(x-T_{j}) (j=1,2,..N)? How can I get the global vectors like x-T_{j} ? PETSc only provides the vector x-T_{j} on the subdomain now. Thanks very much! -- Best wishes, Lulu Liu -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yhcy1993 at gmail.com Sun Nov 7 02:45:45 2021 From: yhcy1993 at gmail.com (=?UTF-8?B?5LuT5a6H?=) Date: Sun, 7 Nov 2021 16:45:45 +0800 Subject: [petsc-users] Strange behavior of TS after setting hand-coded Jacobian In-Reply-To: <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> References: <0C6ACBF3-F457-4BFD-AD19-8C455444748F@petsc.dev> <9CC15214-4601-4554-808F-C3E96DC3D34A@petsc.dev> <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> Message-ID: Thanks for your patient explanation, and sorry for this late reply. I do understand the manipulation skills that you mentioned, but this kind of manually substitution will not change the numerical difficulty of the problem. In fact, before I turn to PETSc, I have tried to form an ODE for this problem already, by utilizing the well-established functions like 'ode15s' and 'ode23' in MATLAB. But it failed to produce physically reasonable results. The key flaw lies in the treatment of the eigenvalue, namely \hat{\Lambda}_r denoted in the PDF file. Only implicit DAE formulation can fully embody its effect. This is why I finally decided to turn to use the DAE functionality in PETSc. I've excerpted some description from a monograph regarding this problem, hope it can clearly demonstrate the numerical requirement for solving it. [image: bvp.png] Kee, R. J., Coltrin, M. E., Glarborg, P., & Zhu, H. (2017). Stagnation Flows. In *Chemically Reacting Flow* (2 ed., pp. 232). Zhang, Hong ?2021?11?1??? ??10:21??? > You can avoid the zero-diagonal elements in your case. Just remove the > boundary points and the \hat{\Lambda}_r from the solution vector Y (eq(17) > in your PDE file). You can calculate them separately in your RHS function > (when computing the residual vector for 2 <=i<=N-1). It is not necessary to > form a DAE. You can actually form an ODE instead. > > Hong (Mr.) > > > On Oct 31, 2021, at 5:38 AM, ?? wrote: > > > > Thanks for your suggestions and experience sharing. > > > > But it seems that you have missed my point. This kind of zero-diagonal > > element that I described earlier is somehow inevitable. > > If interested in this problem, you may refer to the PDF file that was > > attached in the previous post. > > > > Regards > > > > Yu > > > > Zhang, Hong ?2021?10?29??? ??10:05??? > >> > >> One way to avoid the zero element in Jacobian is to exclude the > boundary point from the solution vector. I often do this for Dirichlet > boundary conditions since the value at the boundary is given directly and > does not need to be taken as a degree of freedom. > >> > >> Hong (Mr.) > >> > >> On Oct 28, 2021, at 9:49 PM, ?? wrote: > >> > >> Thanks for your careful inspection and thoughtful suggestions. > >> > >>> finite differencing may put a small non-zero value in that location > due to numerical round-off > >> > >> I think your explanation is reasonable. This numerical round-off may > somehow help to avoid this pivot issue. > >> > >> The structure of my jacobian matrix looks like this (generated by > '-mat_view draw'): > >> > >> Analytically, the first diagonal element of the jacobian is indeed 0, > as its corresponding residual function is solely determined from boundary > condition of another variable. This seems a little bit wired but is > mathematically well-posed. For more description about the background > physics, please refer to attached PDF file for more detailed explanation on > the discretization and boundary conditions. > >> > >> Actually, the jacobian matrix is not singular, but I do believe this > numerical difficulty is caused by the zero-element in diagonal. > >> In this regard, I've performed some trial and test. It seems that > several methods have been worked out for this pivot issue: > >> a) By setting '-pc_type svd', PETSC does not panic any more with my > hand-coded jacobian, and converged solution is obtained. Efficiency is also > preserved. > >> b) By setting '-pc_type none', converged solution is also obtained, but > it takes too many KSP iterations to converge per SNES step (usually > hundreds), making the overall solution procedure very slow. > >> > >> Do you think these methods really solved this kind of pivot issue? Not > by chance like the numerical round-off in finite difference previously. > >> > >> Regards > >> > >> Yu Cang > >> > >> Barry Smith ?2021?10?27??? ??9:43??? > >>> > >>> > >>> You can run with -ksp_error_if_not_converged to get it to stop as > soon as a linear solve fails to help track down the exact breaking point. > >>> > >>>> The problem under consideration contains an eigen-value to be solved, > >>>> making the first diagonal element of the jacobian matrix being zero. > >>>> From these outputs, it seems that the PC failed to factorize, which is > >>>> caused by this 0 diagonal element. But I'm wondering why it works > >>>> with jacobian matrix generated by finite-difference? > >>> > >>> Presumably your "exact" Jacobian puts a zero on the diagonal while > the finite differencing may put a small non-zero value in that location due > to numerical round-off. In that case even if the factorization succeeds it > may produce an inaccurate solution if the value on the diagonal is very > small. > >>> > >>> If your matrix is singular or cannot be factored with LU then you > need to use a different solver for the linear system that will be robust to > the zero on the diagonal. What is the structure of your Jacobian? (The > analytic form). > >>> > >>> Barry > >>> > >>> > >>>> On Oct 27, 2021, at 1:47 AM, ?? wrote: > >>>> > >>>> Thanks for your kind reply. > >>>> > >>>> Several comparison tests have been performed. Attached are execution > >>>> output files. Below are corresponding descriptions. > >>>> > >>>> good.txt -- Run without hand-coded jacobian, solution converged, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason'; > >>>> jac1.txt -- Run with hand-coded jacobian, does not converge, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian'; > >>>> jac2.txt -- Run with hand-coded jacobian, does not converge, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian > >>>> -ksp_view'; > >>>> jac3.txt -- Run with hand-coded jacobian, does not converge, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian > >>>> -ksp_view -ts_max_snes_failures -1 '; > >>>> > >>>> The problem under consideration contains an eigen-value to be solved, > >>>> making the first diagonal element of the jacobian matrix being zero. > >>>> From these outputs, it seems that the PC failed to factorize, which is > >>>> caused by this 0 diagonal element. But I'm wondering why it works > >>>> with jacobian matrix generated by finite-difference? Would employing > >>>> DMDA for discretization be helpful? > >>>> > >>>> Regards > >>>> > >>>> Yu Cang > >>>> > >>>> Barry Smith ?2021?10?25??? ??10:50??? > >>>>> > >>>>> > >>>>> It is definitely unexpected that -snes_test_jacobian verifies the > Jacobian as matching but the solve process is completely different. > >>>>> > >>>>> Please run with -snes_monitor -snes_converged_reason > -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian and > send all the output > >>>>> > >>>>> Barry > >>>>> > >>>>> > >>>>>> On Oct 25, 2021, at 9:53 AM, ?? wrote: > >>>>>> > >>>>>> I'm using TS to solve a set of DAE, which originates from a > >>>>>> one-dimensional problem. The grid points are uniformly distributed. > >>>>>> For simplicity, the DMDA is not employed for discretization. > >>>>>> > >>>>>> At first, only the residual function is prescribed through > >>>>>> 'TSSetIFunction', and PETSC produces converged results. However, > after > >>>>>> providing hand-coded Jacobian through 'TSSetIJacobian', the internal > >>>>>> SNES object fails (residual norm does not change), and TS reports > >>>>>> 'DIVERGED_STEP_REJECTED'. > >>>>>> > >>>>>> I have tried to add the option '-snes_test_jacobian' to see if the > >>>>>> hand-coded jacobian is somewhere wrong, but it shows '||J - > >>>>>> Jfd||_F/||J||_F = 1.07488e-10, ||J - Jfd||_F = 2.14458e-07', > >>>>>> indicating that the hand-coded jacobian is correct. > >>>>>> > >>>>>> Then, I added a monitor for the internal SNES object through > >>>>>> 'SNESMonitorSet', in which the solution vector will be displayed at > >>>>>> each iteration. It is interesting to find that, if the jacobian is > not > >>>>>> provided, meaning finite-difference is utilized for jacobian > >>>>>> evaluation internally, the solution vector converges to steady > >>>>>> solution and the SNES residual norm is reduced continuously. > However, > >>>>>> it turns out that, as long as the jacobian is provided, the solution > >>>>>> vector will NEVER get changed! So the solution procedure stucked! > >>>>>> > >>>>>> This is quite strange! Hope to get some advice. > >>>>>> PETSC version=3.14.6, program run in serial mode. > >>>>>> > >>>>>> Regards > >>>>>> > >>>>>> Yu Cang > >>>>> > >>>> > >>> > >> > >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: bvp.png Type: image/png Size: 162381 bytes Desc: not available URL: From bsmith at petsc.dev Sun Nov 7 09:14:58 2021 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 7 Nov 2021 10:14:58 -0500 Subject: [petsc-users] Strange behavior of TS after setting hand-coded Jacobian In-Reply-To: References: <0C6ACBF3-F457-4BFD-AD19-8C455444748F@petsc.dev> <9CC15214-4601-4554-808F-C3E96DC3D34A@petsc.dev> <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> Message-ID: I think you can use the following approach. Instead of making the boundary conditions, for example, V_1(t) = 0 you do the equivalent, v_1(t=0) = 0 and d V_1(t) ---------- = 0 dt Now if you do backward Euler on this you get V_1^{n+1} - V_1^{n} -------------------------- = 0 Dt or in your residual form V_1^{n+1} - V_1^{n} R_{V_1} = -------------------------- Dt The Jacobian of this gives a 1/Dt on the diagonal for this row. (Similarly for the other problematic rows). Now your Jacobian is nonsingular and you can solve with it with no difficulties. Note that you get the same result if you think of the problem as a DAE because the residual equation is R_{V_1} = V_1^{n+1} and the derivative of this with respect to V_1^{n+1} is 1 (not zero). So you still end up with a nonzero on the diagonal, not zero. Regardless, you can keep these rows in the ODE form right hand side and form Jacobian, or you can eliminate them (analytically as Hong suggested) so the ODE solver does not see these rows but in either case the results are identical. Barry > On Nov 7, 2021, at 3:45 AM, ?? wrote: > > Thanks for your patient explanation, and sorry for this late reply. > > I do understand the manipulation skills that you mentioned, but this kind of manually substitution will not change the numerical difficulty of the problem. > > In fact, before I turn to PETSc, I have tried to form an ODE for this problem already, by utilizing the well-established functions like 'ode15s' and 'ode23' in MATLAB. But it failed to produce physically reasonable results. The key flaw lies in the treatment of the eigenvalue, namely \hat{\Lambda}_r denoted in the PDF file. Only implicit DAE formulation can fully embody its effect. This is why I finally decided to turn to use the DAE functionality in PETSc. > > I've excerpted some description from a monograph regarding this problem, hope it can clearly demonstrate the numerical requirement for solving it. > > Kee, R. J., Coltrin, M. E., Glarborg, P., & Zhu, H. (2017). Stagnation Flows. In Chemically Reacting Flow (2 ed., pp. 232). > > > > Zhang, Hong > ?2021?11?1??? ??10:21??? > You can avoid the zero-diagonal elements in your case. Just remove the boundary points and the \hat{\Lambda}_r from the solution vector Y (eq(17) in your PDE file). You can calculate them separately in your RHS function (when computing the residual vector for 2 <=i<=N-1). It is not necessary to form a DAE. You can actually form an ODE instead. > > Hong (Mr.) > > > On Oct 31, 2021, at 5:38 AM, ?? > wrote: > > > > Thanks for your suggestions and experience sharing. > > > > But it seems that you have missed my point. This kind of zero-diagonal > > element that I described earlier is somehow inevitable. > > If interested in this problem, you may refer to the PDF file that was > > attached in the previous post. > > > > Regards > > > > Yu > > > > Zhang, Hong > ?2021?10?29??? ??10:05??? > >> > >> One way to avoid the zero element in Jacobian is to exclude the boundary point from the solution vector. I often do this for Dirichlet boundary conditions since the value at the boundary is given directly and does not need to be taken as a degree of freedom. > >> > >> Hong (Mr.) > >> > >> On Oct 28, 2021, at 9:49 PM, ?? > wrote: > >> > >> Thanks for your careful inspection and thoughtful suggestions. > >> > >>> finite differencing may put a small non-zero value in that location due to numerical round-off > >> > >> I think your explanation is reasonable. This numerical round-off may somehow help to avoid this pivot issue. > >> > >> The structure of my jacobian matrix looks like this (generated by '-mat_view draw'): > >> > >> Analytically, the first diagonal element of the jacobian is indeed 0, as its corresponding residual function is solely determined from boundary condition of another variable. This seems a little bit wired but is mathematically well-posed. For more description about the background physics, please refer to attached PDF file for more detailed explanation on the discretization and boundary conditions. > >> > >> Actually, the jacobian matrix is not singular, but I do believe this numerical difficulty is caused by the zero-element in diagonal. > >> In this regard, I've performed some trial and test. It seems that several methods have been worked out for this pivot issue: > >> a) By setting '-pc_type svd', PETSC does not panic any more with my hand-coded jacobian, and converged solution is obtained. Efficiency is also preserved. > >> b) By setting '-pc_type none', converged solution is also obtained, but it takes too many KSP iterations to converge per SNES step (usually hundreds), making the overall solution procedure very slow. > >> > >> Do you think these methods really solved this kind of pivot issue? Not by chance like the numerical round-off in finite difference previously. > >> > >> Regards > >> > >> Yu Cang > >> > >> Barry Smith > ?2021?10?27??? ??9:43??? > >>> > >>> > >>> You can run with -ksp_error_if_not_converged to get it to stop as soon as a linear solve fails to help track down the exact breaking point. > >>> > >>>> The problem under consideration contains an eigen-value to be solved, > >>>> making the first diagonal element of the jacobian matrix being zero. > >>>> From these outputs, it seems that the PC failed to factorize, which is > >>>> caused by this 0 diagonal element. But I'm wondering why it works > >>>> with jacobian matrix generated by finite-difference? > >>> > >>> Presumably your "exact" Jacobian puts a zero on the diagonal while the finite differencing may put a small non-zero value in that location due to numerical round-off. In that case even if the factorization succeeds it may produce an inaccurate solution if the value on the diagonal is very small. > >>> > >>> If your matrix is singular or cannot be factored with LU then you need to use a different solver for the linear system that will be robust to the zero on the diagonal. What is the structure of your Jacobian? (The analytic form). > >>> > >>> Barry > >>> > >>> > >>>> On Oct 27, 2021, at 1:47 AM, ?? > wrote: > >>>> > >>>> Thanks for your kind reply. > >>>> > >>>> Several comparison tests have been performed. Attached are execution > >>>> output files. Below are corresponding descriptions. > >>>> > >>>> good.txt -- Run without hand-coded jacobian, solution converged, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason'; > >>>> jac1.txt -- Run with hand-coded jacobian, does not converge, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian'; > >>>> jac2.txt -- Run with hand-coded jacobian, does not converge, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian > >>>> -ksp_view'; > >>>> jac3.txt -- Run with hand-coded jacobian, does not converge, with > >>>> option '-ts_monitor -snes_monitor -snes_converged_reason > >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian > >>>> -ksp_view -ts_max_snes_failures -1 '; > >>>> > >>>> The problem under consideration contains an eigen-value to be solved, > >>>> making the first diagonal element of the jacobian matrix being zero. > >>>> From these outputs, it seems that the PC failed to factorize, which is > >>>> caused by this 0 diagonal element. But I'm wondering why it works > >>>> with jacobian matrix generated by finite-difference? Would employing > >>>> DMDA for discretization be helpful? > >>>> > >>>> Regards > >>>> > >>>> Yu Cang > >>>> > >>>> Barry Smith > ?2021?10?25??? ??10:50??? > >>>>> > >>>>> > >>>>> It is definitely unexpected that -snes_test_jacobian verifies the Jacobian as matching but the solve process is completely different. > >>>>> > >>>>> Please run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian and send all the output > >>>>> > >>>>> Barry > >>>>> > >>>>> > >>>>>> On Oct 25, 2021, at 9:53 AM, ?? > wrote: > >>>>>> > >>>>>> I'm using TS to solve a set of DAE, which originates from a > >>>>>> one-dimensional problem. The grid points are uniformly distributed. > >>>>>> For simplicity, the DMDA is not employed for discretization. > >>>>>> > >>>>>> At first, only the residual function is prescribed through > >>>>>> 'TSSetIFunction', and PETSC produces converged results. However, after > >>>>>> providing hand-coded Jacobian through 'TSSetIJacobian', the internal > >>>>>> SNES object fails (residual norm does not change), and TS reports > >>>>>> 'DIVERGED_STEP_REJECTED'. > >>>>>> > >>>>>> I have tried to add the option '-snes_test_jacobian' to see if the > >>>>>> hand-coded jacobian is somewhere wrong, but it shows '||J - > >>>>>> Jfd||_F/||J||_F = 1.07488e-10, ||J - Jfd||_F = 2.14458e-07', > >>>>>> indicating that the hand-coded jacobian is correct. > >>>>>> > >>>>>> Then, I added a monitor for the internal SNES object through > >>>>>> 'SNESMonitorSet', in which the solution vector will be displayed at > >>>>>> each iteration. It is interesting to find that, if the jacobian is not > >>>>>> provided, meaning finite-difference is utilized for jacobian > >>>>>> evaluation internally, the solution vector converges to steady > >>>>>> solution and the SNES residual norm is reduced continuously. However, > >>>>>> it turns out that, as long as the jacobian is provided, the solution > >>>>>> vector will NEVER get changed! So the solution procedure stucked! > >>>>>> > >>>>>> This is quite strange! Hope to get some advice. > >>>>>> PETSC version=3.14.6, program run in serial mode. > >>>>>> > >>>>>> Regards > >>>>>> > >>>>>> Yu Cang > >>>>> > >>>> > >>> > >> > >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nabw91 at gmail.com Mon Nov 8 09:59:35 2021 From: nabw91 at gmail.com (barnafi) Date: Mon, 08 Nov 2021 16:59:35 +0100 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel Message-ID: Hi PETSc users, I am testing a block preconditioner for a nonlinear problem using fenics for matrix assembly, and I obtain correcet results in serial only (parallel diverges). The difficulty lies in using Nest matrices, as I want a performant code... I already see astonishing gains in serial. The code seems simple enough, where I have a UFL form from which I obtain all different blocks. Then, I create a SNESSolver as follows: --------------- SNESSolver.py ----------------- class SNESProblem(): def __init__(self, FFs, JJs, uu, pp, bbcs): self.Ls = FFs # UFL form for residuals self.Js = JJs # UFL form for jacobian self.bcs = bbcs self.bcs0 = [DirichletBC(_bc) for _bc in bbcs] # Homogenized boundary conditions for Newton iterations for bc in self.bcs0: bc.homogenize() self.u = uu self.p = pp def assemble_F(self, FF): Fu, Fp = FF.getNestSubVecs() assemble(self.Ls[0], tensor=PETScVector(Fu)) assemble(self.Ls[1], tensor=PETScVector(Fp)) for bc in self.bcs0: bc.apply(PETScVector(Fu)) Fu.assemble() FF.assemble() def assemble_J(self, JJ): Juu = JJ.getNestSubMatrix(0, 0) Jup = JJ.getNestSubMatrix(0, 1) Jpu = JJ.getNestSubMatrix(1, 0) Jpp = JJ.getNestSubMatrix(1, 1) assemble(self.Js[0], tensor=PETScMatrix(Juu)) assemble(self.Js[1], tensor=PETScMatrix(Jup)) assemble(self.Js[2], tensor=PETScMatrix(Jpu)) assemble(self.Js[3], tensor=PETScMatrix(Jpp)) MM = Juu.getDiagonal().sum() / Juu.getSize()[0] # Better performance for bc in self.bcs0: indexes = [*bc.get_boundary_values().keys()] Juu.zeroRows(indexes, diag=MM) Jup.zeroRows(indexes, diag=0.0) Juu.assemble() Jup.assemble() JJ.assemble() def F(self, snes, xx, FF): xu, xp = xx.getNestSubVecs() # Update fenics vectors to keep track of ghost nodes xu.copy(self.u.vector().vec()) xp.copy(self.p.vector().vec()) self.u.vector().apply("") self.p.vector().apply("") self.assemble_F(FF) def J(self, snes, xx, JJ, PP): xu, xp = xx.getNestSubVecs() xu.copy(self.u.vector().vec()) xp.copy(self.p.vector().vec()) # Update fenics vectors to keep track of ghost nodes self.u.vector().apply("") self.p.vector().apply("") self.assemble_J(JJ) ----------- SNESSolver.py end ------------- I then initialize the Jacobian matrices to create the Nest (no 'isrows' nor 'iscols') and then initialize the PC from the SNES with ------ PC init ------ is_0, is_1 = J_mat.getNestISs()[0] ksp = snes.getKSP() pc = ksp.getPC() pc.setType("fieldsplit") pc.setFieldSplitIS(("0", is_0), ("1", is_1)) --------------------- The current behavior is that in serial everything works as expected, and instead in parallel not even the boundary conditions are set up correctly. The two possible problems I see regard the ordering of the dofs, or the update of ghost nodes, but I have tried modifying the code to account for this without success. If anyone has had similar issues, I would appreciate any comments. Thanks in advance. Best, NB From daniel.stone at opengosim.com Mon Nov 8 13:24:07 2021 From: daniel.stone at opengosim.com (Daniel Stone) Date: Mon, 8 Nov 2021 19:24:07 +0000 Subject: [petsc-users] Configure of 3.16.1 failing on mpich/yaksa Message-ID: Hello all, I've been having some configure failures trying to configure petsc, on Ubuntu 20, when downloading mpich. This seems to be related to the use of "#!/bin/sh" found in the script mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh /bin/sh in Ubuntu20 is dash, not bash, and line 35 of the script is: CMD="${@:2} -Xcompiler -fPIC -o $PIC_FILEPATH" which is apparently not valid dash syntax. I see "bad substitution" errors when trying to run this script in isolation, which can be fixed by replacing the top line with "#!/bin/bash" The petsc config log points to this line in this script: ------------------------------------------------------------------------------------ make[2]: Entering directory '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hindexed__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_contig__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hvector__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_resized__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_blkhindx__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hindexed__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_contig__Bool.lo NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_resized__Bool.lo make[2]: Leaving directory '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' make[1]: Leaving directory '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2'/usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution ./src/backend/cuda/cudalt.sh: 35: Bad substitution make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 make[2]: *** Waiting for unfinished jobs.... make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo] Error 2 make[2]: *** [Makefile:8697: src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo] Error 2 ---------------------------------------------------------------------------- What is interesting is the choice made by the config script here to make yaksa "cuda-aware", which I do not understand how to control. By this I mean - the use of NVCC, the use of files with "cudai" in the name, and the running of the cudalt.sh script. This is especially odd given that on another machine, also with Ubuntu20, none of this occurs, despite using the exact same configure instructions: --------------------------------------------------- CC src/backend/seq/pup/yaksuri_seqi_pup_blkhindx_resized__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hvector__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_blkhindx__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hindexed__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_contig__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_resized__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_contig__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hvector__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_contig_blkhindx__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hindexed__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_contig_contig__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_contig_resized__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_resized__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hvector__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_resized_blkhindx__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hindexed__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_resized_contig__Bool.lo CC src/backend/seq/pup/yaksuri_seqi_pup_resized_resized__Bool.lo ------------------------------------------------------- On both machines petsc locates a working nvcc without trouble. One difference is that on the good machine, cuda 11 is installed, built from source, while on the bad machine cuda 10 is installed, installed via apt-get. On the good machine, running the cudalt.sh script in isolation results in the same bad substitution error. Can someone help me understand where the difference of behaviour might come from, w.r.t. cuda? Adding the --with-cuda=0 flag on the bad machine made no difference. Is there any way of communicating to mpich/yaksuri that I don't want whatever features that involve the cudalt.sh script being run? The configure command I use is: ./configure --download-mpich=yes --download-hdf5=yes --download-fblaslapack=yes --download-metis=yes --download-cmake=yes --download-ptscotch=yes --download-hypre=yes --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 -download-hdf5-fortran-bindings=yes --download-sowing Thanks, Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Nov 8 13:35:53 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 8 Nov 2021 13:35:53 -0600 (CST) Subject: [petsc-users] Configure of 3.16.1 failing on mpich/yaksa In-Reply-To: References: Message-ID: <801382a1-a618-dcf9-e7da-a868343344d@mcs.anl.gov> This looks like a bug report to mpich. You should be able to reproduce this without PETSc - by directly building MPICH. And then report to MPICH developers. Wrt petsc - you could try --download-openmpi [instead of --download-mpich] and see if that works better. Yeah cuda obtained from nvida and cuda repackaged by ubuntu have subtle differences that can cause cuda failures. As you say - mpich might have a configure option to disable this. If you are able to find this option - you can use it via petsc configure with: --download-mpich-configure-arguments=string Satish On Mon, 8 Nov 2021, Daniel Stone wrote: > Hello all, > > I've been having some configure failures trying to configure petsc, on > Ubuntu 20, when > downloading mpich. > > > This seems to be related to the use of > "#!/bin/sh" > found in the script > mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh > > /bin/sh in Ubuntu20 is dash, not bash, and line 35 of the script is: > CMD="${@:2} -Xcompiler -fPIC -o $PIC_FILEPATH" > which is apparently not valid dash syntax. I see "bad substitution" errors > when > trying to run this script in isolation, which can be fixed by replacing the > top line with > > "#!/bin/bash" > > The petsc config log points to this line in this script: > > ------------------------------------------------------------------------------------ > > make[2]: Entering directory > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hindexed__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_contig__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hvector__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_resized__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_blkhindx__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hindexed__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_contig__Bool.lo > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_resized__Bool.lo > make[2]: Leaving directory > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > make[1]: Leaving directory > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2'/usr/bin/ar: `u' > modifier ignored since `D' is the default (see `U') > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > make[2]: *** [Makefile:8697: > src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 > make[2]: *** Waiting for unfinished jobs.... > make[2]: *** [Makefile:8697: > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo] Error 2 > make[2]: *** [Makefile:8697: > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo] Error 2 > > ---------------------------------------------------------------------------- > > What is interesting is the choice made by the config script here to make > yaksa "cuda-aware", which I do not > understand how to control. By this I mean - the use of NVCC, the use of > files with "cudai" in the name, > and the running of the cudalt.sh script. > > This is especially odd given that on another machine, also with Ubuntu20, > none of this occurs, despite > using the exact same configure instructions: > > --------------------------------------------------- > CC src/backend/seq/pup/yaksuri_seqi_pup_blkhindx_resized__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hvector__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_blkhindx__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hindexed__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_contig__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_resized__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_contig__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hvector__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_blkhindx__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hindexed__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_contig__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_resized__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_resized__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hvector__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_blkhindx__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hindexed__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_contig__Bool.lo > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_resized__Bool.lo > > > ------------------------------------------------------- > > On both machines petsc locates a working nvcc without trouble. One > difference is > that on the good machine, cuda 11 is installed, built from source, while on > the bad machine > cuda 10 is installed, installed via apt-get. On the good machine, running > the cudalt.sh script in > isolation results in the same bad substitution error. > > Can someone help me understand where the difference of behaviour might come > from, w.r.t. > cuda? Adding the --with-cuda=0 flag on the bad machine made no difference. > Is there any way > of communicating to mpich/yaksuri that I don't want whatever features that > involve the cudalt.sh > script being run? > > The configure command I use is: > > ./configure --download-mpich=yes --download-hdf5=yes > --download-fblaslapack=yes --download-metis=yes --download-cmake=yes > --download-ptscotch=yes --download-hypre=yes --with-debugging=0 > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 > -download-hdf5-fortran-bindings=yes --download-sowing > > > > Thanks, > > Daniel > From balay at mcs.anl.gov Mon Nov 8 13:41:05 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 8 Nov 2021 13:41:05 -0600 (CST) Subject: [petsc-users] Configure of 3.16.1 failing on mpich/yaksa In-Reply-To: <801382a1-a618-dcf9-e7da-a868343344d@mcs.anl.gov> References: <801382a1-a618-dcf9-e7da-a868343344d@mcs.anl.gov> Message-ID: Alternatively you can try an older MPICH release - perhaps this issue is related to a newly introduced feature. --download-mpich=https://www.mpich.org/static/downloads/3.3.2/mpich-3.3.2.tar.gz Satish On Mon, 8 Nov 2021, Satish Balay via petsc-users wrote: > This looks like a bug report to mpich. > > You should be able to reproduce this without PETSc - by directly building MPICH. > > And then report to MPICH developers. > > Wrt petsc - you could try --download-openmpi [instead of --download-mpich] and see if that works better. > > Yeah cuda obtained from nvida and cuda repackaged by ubuntu have subtle differences that can cause cuda failures. > > As you say - mpich might have a configure option to disable this. If you are able to find this option - you can use it via petsc configure with: > > --download-mpich-configure-arguments=string > > Satish > > On Mon, 8 Nov 2021, Daniel Stone wrote: > > > Hello all, > > > > I've been having some configure failures trying to configure petsc, on > > Ubuntu 20, when > > downloading mpich. > > > > > > This seems to be related to the use of > > "#!/bin/sh" > > found in the script > > mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh > > > > /bin/sh in Ubuntu20 is dash, not bash, and line 35 of the script is: > > CMD="${@:2} -Xcompiler -fPIC -o $PIC_FILEPATH" > > which is apparently not valid dash syntax. I see "bad substitution" errors > > when > > trying to run this script in isolation, which can be fixed by replacing the > > top line with > > > > "#!/bin/bash" > > > > The petsc config log points to this line in this script: > > > > ------------------------------------------------------------------------------------ > > > > make[2]: Entering directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hindexed__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_contig__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hvector__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_resized__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_blkhindx__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hindexed__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_contig__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_resized__Bool.lo > > make[2]: Leaving directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > > make[1]: Leaving directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2'/usr/bin/ar: `u' > > modifier ignored since `D' is the default (see `U') > > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 > > make[2]: *** Waiting for unfinished jobs.... > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo] Error 2 > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo] Error 2 > > > > ---------------------------------------------------------------------------- > > > > What is interesting is the choice made by the config script here to make > > yaksa "cuda-aware", which I do not > > understand how to control. By this I mean - the use of NVCC, the use of > > files with "cudai" in the name, > > and the running of the cudalt.sh script. > > > > This is especially odd given that on another machine, also with Ubuntu20, > > none of this occurs, despite > > using the exact same configure instructions: > > > > --------------------------------------------------- > > CC src/backend/seq/pup/yaksuri_seqi_pup_blkhindx_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hvector__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_blkhindx__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hvector__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_blkhindx__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hvector__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_blkhindx__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_resized__Bool.lo > > > > > > ------------------------------------------------------- > > > > On both machines petsc locates a working nvcc without trouble. One > > difference is > > that on the good machine, cuda 11 is installed, built from source, while on > > the bad machine > > cuda 10 is installed, installed via apt-get. On the good machine, running > > the cudalt.sh script in > > isolation results in the same bad substitution error. > > > > Can someone help me understand where the difference of behaviour might come > > from, w.r.t. > > cuda? Adding the --with-cuda=0 flag on the bad machine made no difference. > > Is there any way > > of communicating to mpich/yaksuri that I don't want whatever features that > > involve the cudalt.sh > > script being run? > > > > The configure command I use is: > > > > ./configure --download-mpich=yes --download-hdf5=yes > > --download-fblaslapack=yes --download-metis=yes --download-cmake=yes > > --download-ptscotch=yes --download-hypre=yes --with-debugging=0 > > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 > > -download-hdf5-fortran-bindings=yes --download-sowing > > > > > > > > Thanks, > > > > Daniel > > > From daniel.stone at opengosim.com Mon Nov 8 14:01:44 2021 From: daniel.stone at opengosim.com (Daniel Stone) Date: Mon, 8 Nov 2021 20:01:44 +0000 Subject: [petsc-users] Configure of 3.16.1 failing on mpich/yaksa In-Reply-To: <801382a1-a618-dcf9-e7da-a868343344d@mcs.anl.gov> References: <801382a1-a618-dcf9-e7da-a868343344d@mcs.anl.gov> Message-ID: Aha - they already know. A commit from April in their main repo fixes exactly this problem. It's present in the 4.0.a.2 alpha, but it looks like we have to wait until a new release version with the fix in. https://github.com/pmodels/yaksa/commit/eed193d9775dd0f33cbd8caa0dd946647b751b18#diff-f5310b2c9b83ad225b424c6ab70b970c2c57a4db39daf7b4f8c017df92646c84 On Mon, Nov 8, 2021 at 7:36 PM Satish Balay wrote: > This looks like a bug report to mpich. > > You should be able to reproduce this without PETSc - by directly building > MPICH. > > And then report to MPICH developers. > > Wrt petsc - you could try --download-openmpi [instead of --download-mpich] > and see if that works better. > > Yeah cuda obtained from nvida and cuda repackaged by ubuntu have subtle > differences that can cause cuda failures. > > As you say - mpich might have a configure option to disable this. If you > are able to find this option - you can use it via petsc configure with: > > --download-mpich-configure-arguments=string > > Satish > > On Mon, 8 Nov 2021, Daniel Stone wrote: > > > Hello all, > > > > I've been having some configure failures trying to configure petsc, on > > Ubuntu 20, when > > downloading mpich. > > > > > > This seems to be related to the use of > > "#!/bin/sh" > > found in the script > > mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh > > > > /bin/sh in Ubuntu20 is dash, not bash, and line 35 of the script is: > > CMD="${@:2} -Xcompiler -fPIC -o $PIC_FILEPATH" > > which is apparently not valid dash syntax. I see "bad substitution" > errors > > when > > trying to run this script in isolation, which can be fixed by replacing > the > > top line with > > > > "#!/bin/bash" > > > > The petsc config log points to this line in this script: > > > > > ------------------------------------------------------------------------------------ > > > > make[2]: Entering directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hindexed__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_contig__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hvector__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_resized__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_blkhindx__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hindexed__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_contig__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_resized__Bool.lo > > make[2]: Leaving directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > > make[1]: Leaving directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2'/usr/bin/ar: > `u' > > modifier ignored since `D' is the default (see `U') > > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 > > make[2]: *** Waiting for unfinished jobs.... > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo] Error 2 > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo] Error 2 > > > > > ---------------------------------------------------------------------------- > > > > What is interesting is the choice made by the config script here to make > > yaksa "cuda-aware", which I do not > > understand how to control. By this I mean - the use of NVCC, the use of > > files with "cudai" in the name, > > and the running of the cudalt.sh script. > > > > This is especially odd given that on another machine, also with Ubuntu20, > > none of this occurs, despite > > using the exact same configure instructions: > > > > --------------------------------------------------- > > CC src/backend/seq/pup/yaksuri_seqi_pup_blkhindx_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hvector__Bool.lo > > CC > src/backend/seq/pup/yaksuri_seqi_pup_hindexed_blkhindx__Bool.lo > > CC > src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hvector__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_blkhindx__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hvector__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_blkhindx__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_resized__Bool.lo > > > > > > ------------------------------------------------------- > > > > On both machines petsc locates a working nvcc without trouble. One > > difference is > > that on the good machine, cuda 11 is installed, built from source, while > on > > the bad machine > > cuda 10 is installed, installed via apt-get. On the good machine, running > > the cudalt.sh script in > > isolation results in the same bad substitution error. > > > > Can someone help me understand where the difference of behaviour might > come > > from, w.r.t. > > cuda? Adding the --with-cuda=0 flag on the bad machine made no > difference. > > Is there any way > > of communicating to mpich/yaksuri that I don't want whatever features > that > > involve the cudalt.sh > > script being run? > > > > The configure command I use is: > > > > ./configure --download-mpich=yes --download-hdf5=yes > > --download-fblaslapack=yes --download-metis=yes --download-cmake=yes > > --download-ptscotch=yes --download-hypre=yes --with-debugging=0 > > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 > > -download-hdf5-fortran-bindings=yes --download-sowing > > > > > > > > Thanks, > > > > Daniel > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 8 14:02:42 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 8 Nov 2021 15:02:42 -0500 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: References: Message-ID: On Mon, Nov 8, 2021 at 11:00 AM barnafi wrote: > Hi PETSc users, > > I am testing a block preconditioner for a nonlinear problem using > fenics for matrix assembly, and I obtain correcet results in serial > only (parallel diverges). The difficulty lies in using Nest matrices, > as I want a performant code... I already see astonishing gains in > serial. > > The code seems simple enough, where I have a UFL form from which I > obtain all different blocks. Then, I create a SNESSolver as follows: > > > --------------- SNESSolver.py ----------------- > class SNESProblem(): > def __init__(self, FFs, JJs, uu, pp, bbcs): > self.Ls = FFs # UFL form for residuals > self.Js = JJs # UFL form for jacobian > self.bcs = bbcs > self.bcs0 = [DirichletBC(_bc) for _bc in bbcs] # Homogenized > boundary conditions for Newton iterations > for bc in self.bcs0: > bc.homogenize() > self.u = uu > self.p = pp > > def assemble_F(self, FF): > Fu, Fp = FF.getNestSubVecs() > assemble(self.Ls[0], tensor=PETScVector(Fu)) > assemble(self.Ls[1], tensor=PETScVector(Fp)) > for bc in self.bcs0: > bc.apply(PETScVector(Fu)) > Fu.assemble() > FF.assemble() > > def assemble_J(self, JJ): > Juu = JJ.getNestSubMatrix(0, 0) > Jup = JJ.getNestSubMatrix(0, 1) > Jpu = JJ.getNestSubMatrix(1, 0) > Jpp = JJ.getNestSubMatrix(1, 1) > assemble(self.Js[0], tensor=PETScMatrix(Juu)) > assemble(self.Js[1], tensor=PETScMatrix(Jup)) > assemble(self.Js[2], tensor=PETScMatrix(Jpu)) > assemble(self.Js[3], tensor=PETScMatrix(Jpp)) > MM = Juu.getDiagonal().sum() / Juu.getSize()[0] # Better > performance > for bc in self.bcs0: > indexes = [*bc.get_boundary_values().keys()] > Juu.zeroRows(indexes, diag=MM) > Jup.zeroRows(indexes, diag=0.0) > Juu.assemble() > Jup.assemble() > JJ.assemble() > > def F(self, snes, xx, FF): > > xu, xp = xx.getNestSubVecs() > # Update fenics vectors to keep track of ghost nodes > xu.copy(self.u.vector().vec()) > xp.copy(self.p.vector().vec()) > self.u.vector().apply("") > self.p.vector().apply("") > self.assemble_F(FF) > > def J(self, snes, xx, JJ, PP): > xu, xp = xx.getNestSubVecs() > xu.copy(self.u.vector().vec()) > xp.copy(self.p.vector().vec()) > # Update fenics vectors to keep track of ghost nodes > self.u.vector().apply("") > self.p.vector().apply("") > self.assemble_J(JJ) > ----------- SNESSolver.py end ------------- > > I then initialize the Jacobian matrices to create the Nest (no 'isrows' > nor 'iscols') and then initialize the PC from the SNES with > > ------ PC init ------ > is_0, is_1 = J_mat.getNestISs()[0] > ksp = snes.getKSP() > pc = ksp.getPC() > pc.setType("fieldsplit") > pc.setFieldSplitIS(("0", is_0), ("1", is_1)) > --------------------- > > The current behavior is that in serial everything works as expected, > and instead in parallel not even the boundary conditions are set up > correctly. The two possible problems I see regard the ordering of the > dofs, or the update of ghost nodes, but I have tried modifying the code > to account for this without success. > > If anyone has had similar issues, I would appreciate any comments. > I am not adept enough at FEniCS to look at the code and debug. However, I would make a very small problem for 2 processes, say two cells on each, and then print out is_0 and is_1. That should tell you if you have the right unknowns in each block. If so, then it seems like something is not correct in your assembly routine, which is probably better debugged on the FEniCS list. However, if they are not right, we can help get them right. THanks, Matt > Thanks in advance. > > Best, > NB > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Mon Nov 8 16:38:21 2021 From: jed at jedbrown.org (Jed Brown) Date: Mon, 08 Nov 2021 15:38:21 -0700 Subject: [petsc-users] Strange behavior of TS after setting hand-coded Jacobian In-Reply-To: References: <0C6ACBF3-F457-4BFD-AD19-8C455444748F@petsc.dev> <9CC15214-4601-4554-808F-C3E96DC3D34A@petsc.dev> <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> Message-ID: <871r3quvc2.fsf@jedbrown.org> Barry Smith writes: > Note that you get the same result if you think of the problem as a DAE because the residual equation is R_{V_1} = V_1^{n+1} and the derivative of this with respect to V_1^{n+1} is 1 (not zero). So you still end up with a nonzero on the diagonal, not zero. Regardless, you can keep these rows in the ODE form right hand side and form Jacobian, or you can eliminate them (analytically as Hong suggested) so the ODE solver does not see these rows but in either case the results are identical. This is a niche technical point, but these strategies (DAE versus ODE enforcement of boundary conditions) are not always equivalent. In particular, the DAE form can exhibit order reduction, meaning that you may observe a lower order of accuracy with this form of boundary conditions than with the ODE form. From ynarayan at buffalo.edu Mon Nov 8 20:51:36 2021 From: ynarayan at buffalo.edu (Yaswanth Narayana) Date: Tue, 9 Nov 2021 02:51:36 +0000 Subject: [petsc-users] Problem with using make Message-ID: Hello My name is Yaswanth Narayana. I am using PETSc as part of my course HPC1. When trying to use make I am getting the following error. What should I do? makefile:8: /lib/petsc/conf/rules: No such file or directory make: *** No rule to make target 'PETSC_DIR/lib/petsc/conf/rules'. Stop. I have tried changing the directory name and changed to the directory containing files 'rules' and 'variables' but still receiving the same error. Please help in this regard. Thank you Yours sincerely Yaswanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Nov 8 21:16:42 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 8 Nov 2021 22:16:42 -0500 Subject: [petsc-users] Problem with using make In-Reply-To: References: Message-ID: <113248C5-425E-4140-8255-51304A5F8A52@petsc.dev> It seems you missed $ in front of $PETSC_DIR somewhere along the line. I suggest deleting the PETSc directory completely, recloning PETSc and follow the instructions at https://petsc.org/release/install/ very careful to insure there are no typos or missing symbols. Barry > On Nov 8, 2021, at 9:51 PM, Yaswanth Narayana wrote: > > Hello > > My name is Yaswanth Narayana. I am using PETSc as part of my course HPC1. When trying to use make I am getting the following error. What should I do? > > makefile:8: /lib/petsc/conf/rules: No such file or directory > make: *** No rule to make target 'PETSC_DIR/lib/petsc/conf/rules'. Stop. > > I have tried changing the directory name and changed to the directory containing files 'rules' and 'variables' but still receiving the same error. Please help in this regard. > > Thank you > > Yours sincerely > Yaswanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Nov 8 21:18:41 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 8 Nov 2021 22:18:41 -0500 Subject: [petsc-users] The exact Jacobian for ASPIN In-Reply-To: References: Message-ID: <36253D17-53C5-4CCB-B969-3AB698B4822A@petsc.dev> Lulu, Sorry for not responding quicker. This question will take a little research to figure out exactly what is needed, I am not sure it is possible. I note below you have a J_{I} is that intentional? If so what does it mean, or is a typo that should be J_{j} as in the next equation? Barry > On Nov 6, 2021, at 7:19 AM, Lulu Liu wrote: > > Hi, > For ASPIN, the local problem F_{j}(x-T_{j})=0 (j=1,2,...N) is solved. The exact Jacobian of the preconditioned Jacobian is > J_exact=\sum_{j=1}^{N}J_{I}(x-T_{i})J(x-T_{j}). > > It seems that PETSc uses the approximate Jacobian, > J_exact=\sum_{j=1}^{N}J_{j}(x-T_{i})J(x-\sum_{j=1}^{N}T_{j}). > > I want to implement RASPEN, which requires the exact Jacobian of ASPIN. Is there any easy way to compute J(x-T_{j}) (j=1,2,..N)? How can I get the global vectors like x-T_{j} ? PETSc only provides the vector x-T_{j} on the subdomain now. > > Thanks very much! > > -- > Best wishes, > Lulu Liu > > > This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Mon Nov 8 22:59:23 2021 From: hongzhang at anl.gov (Zhang, Hong) Date: Tue, 9 Nov 2021 04:59:23 +0000 Subject: [petsc-users] Strange behavior of TS after setting hand-coded Jacobian In-Reply-To: References: <0C6ACBF3-F457-4BFD-AD19-8C455444748F@petsc.dev> <9CC15214-4601-4554-808F-C3E96DC3D34A@petsc.dev> <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> Message-ID: You are essentially solving an index-2 DAE in the form: dx/dt = f(x,y) 0 = g(x) Note that the solvers you have tried, including 'ode15s' and 'ode23' in MATLAB and PETSc TS solvers, are all designed for ODEs or index-1 DAEs, and they cannot be used to solve index-2 DAEs directly. In practice, you need techniques to lower the index of DAEs first. A textbook approach to solve your problem is to differentiate the algebraic equation twice. Taking d/dt for the second equation gives dg/dx dx/dt = 0 ==> dg/dx f(x,y) = 0 Taking d/dt again gives dg/dx (df/dx f + df/dy dy/dt) = 0 Assuming dg/dx df/dy is not singular, we get dy/dt = -(dg/dx df/dy)^{-1} dg/dx (df/dx f) Now you can solve an ODE system instead of the original DAE. There may exist other approaches for solving this particular problem. But you need to look in the literature. Hong (Mr.) On Nov 7, 2021, at 2:45 AM, ?? > wrote: Thanks for your patient explanation, and sorry for this late reply. I do understand the manipulation skills that you mentioned, but this kind of manually substitution will not change the numerical difficulty of the problem. In fact, before I turn to PETSc, I have tried to form an ODE for this problem already, by utilizing the well-established functions like 'ode15s' and 'ode23' in MATLAB. But it failed to produce physically reasonable results. The key flaw lies in the treatment of the eigenvalue, namely \hat{\Lambda}_r denoted in the PDF file. Only implicit DAE formulation can fully embody its effect. This is why I finally decided to turn to use the DAE functionality in PETSc. I've excerpted some description from a monograph regarding this problem, hope it can clearly demonstrate the numerical requirement for solving it. Kee, R. J., Coltrin, M. E., Glarborg, P., & Zhu, H. (2017). Stagnation Flows. In Chemically Reacting Flow (2 ed., pp. 232). Zhang, Hong > ?2021?11?1??? ??10:21??? You can avoid the zero-diagonal elements in your case. Just remove the boundary points and the \hat{\Lambda}_r from the solution vector Y (eq(17) in your PDE file). You can calculate them separately in your RHS function (when computing the residual vector for 2 <=i<=N-1). It is not necessary to form a DAE. You can actually form an ODE instead. Hong (Mr.) > On Oct 31, 2021, at 5:38 AM, ?? > wrote: > > Thanks for your suggestions and experience sharing. > > But it seems that you have missed my point. This kind of zero-diagonal > element that I described earlier is somehow inevitable. > If interested in this problem, you may refer to the PDF file that was > attached in the previous post. > > Regards > > Yu > > Zhang, Hong > ?2021?10?29??? ??10:05??? >> >> One way to avoid the zero element in Jacobian is to exclude the boundary point from the solution vector. I often do this for Dirichlet boundary conditions since the value at the boundary is given directly and does not need to be taken as a degree of freedom. >> >> Hong (Mr.) >> >> On Oct 28, 2021, at 9:49 PM, ?? > wrote: >> >> Thanks for your careful inspection and thoughtful suggestions. >> >>> finite differencing may put a small non-zero value in that location due to numerical round-off >> >> I think your explanation is reasonable. This numerical round-off may somehow help to avoid this pivot issue. >> >> The structure of my jacobian matrix looks like this (generated by '-mat_view draw'): >> >> Analytically, the first diagonal element of the jacobian is indeed 0, as its corresponding residual function is solely determined from boundary condition of another variable. This seems a little bit wired but is mathematically well-posed. For more description about the background physics, please refer to attached PDF file for more detailed explanation on the discretization and boundary conditions. >> >> Actually, the jacobian matrix is not singular, but I do believe this numerical difficulty is caused by the zero-element in diagonal. >> In this regard, I've performed some trial and test. It seems that several methods have been worked out for this pivot issue: >> a) By setting '-pc_type svd', PETSC does not panic any more with my hand-coded jacobian, and converged solution is obtained. Efficiency is also preserved. >> b) By setting '-pc_type none', converged solution is also obtained, but it takes too many KSP iterations to converge per SNES step (usually hundreds), making the overall solution procedure very slow. >> >> Do you think these methods really solved this kind of pivot issue? Not by chance like the numerical round-off in finite difference previously. >> >> Regards >> >> Yu Cang >> >> Barry Smith > ?2021?10?27??? ??9:43??? >>> >>> >>> You can run with -ksp_error_if_not_converged to get it to stop as soon as a linear solve fails to help track down the exact breaking point. >>> >>>> The problem under consideration contains an eigen-value to be solved, >>>> making the first diagonal element of the jacobian matrix being zero. >>>> From these outputs, it seems that the PC failed to factorize, which is >>>> caused by this 0 diagonal element. But I'm wondering why it works >>>> with jacobian matrix generated by finite-difference? >>> >>> Presumably your "exact" Jacobian puts a zero on the diagonal while the finite differencing may put a small non-zero value in that location due to numerical round-off. In that case even if the factorization succeeds it may produce an inaccurate solution if the value on the diagonal is very small. >>> >>> If your matrix is singular or cannot be factored with LU then you need to use a different solver for the linear system that will be robust to the zero on the diagonal. What is the structure of your Jacobian? (The analytic form). >>> >>> Barry >>> >>> >>>> On Oct 27, 2021, at 1:47 AM, ?? > wrote: >>>> >>>> Thanks for your kind reply. >>>> >>>> Several comparison tests have been performed. Attached are execution >>>> output files. Below are corresponding descriptions. >>>> >>>> good.txt -- Run without hand-coded jacobian, solution converged, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason'; >>>> jac1.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian'; >>>> jac2.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >>>> -ksp_view'; >>>> jac3.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >>>> -ksp_view -ts_max_snes_failures -1 '; >>>> >>>> The problem under consideration contains an eigen-value to be solved, >>>> making the first diagonal element of the jacobian matrix being zero. >>>> From these outputs, it seems that the PC failed to factorize, which is >>>> caused by this 0 diagonal element. But I'm wondering why it works >>>> with jacobian matrix generated by finite-difference? Would employing >>>> DMDA for discretization be helpful? >>>> >>>> Regards >>>> >>>> Yu Cang >>>> >>>> Barry Smith > ?2021?10?25??? ??10:50??? >>>>> >>>>> >>>>> It is definitely unexpected that -snes_test_jacobian verifies the Jacobian as matching but the solve process is completely different. >>>>> >>>>> Please run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian and send all the output >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Oct 25, 2021, at 9:53 AM, ?? > wrote: >>>>>> >>>>>> I'm using TS to solve a set of DAE, which originates from a >>>>>> one-dimensional problem. The grid points are uniformly distributed. >>>>>> For simplicity, the DMDA is not employed for discretization. >>>>>> >>>>>> At first, only the residual function is prescribed through >>>>>> 'TSSetIFunction', and PETSC produces converged results. However, after >>>>>> providing hand-coded Jacobian through 'TSSetIJacobian', the internal >>>>>> SNES object fails (residual norm does not change), and TS reports >>>>>> 'DIVERGED_STEP_REJECTED'. >>>>>> >>>>>> I have tried to add the option '-snes_test_jacobian' to see if the >>>>>> hand-coded jacobian is somewhere wrong, but it shows '||J - >>>>>> Jfd||_F/||J||_F = 1.07488e-10, ||J - Jfd||_F = 2.14458e-07', >>>>>> indicating that the hand-coded jacobian is correct. >>>>>> >>>>>> Then, I added a monitor for the internal SNES object through >>>>>> 'SNESMonitorSet', in which the solution vector will be displayed at >>>>>> each iteration. It is interesting to find that, if the jacobian is not >>>>>> provided, meaning finite-difference is utilized for jacobian >>>>>> evaluation internally, the solution vector converges to steady >>>>>> solution and the SNES residual norm is reduced continuously. However, >>>>>> it turns out that, as long as the jacobian is provided, the solution >>>>>> vector will NEVER get changed! So the solution procedure stucked! >>>>>> >>>>>> This is quite strange! Hope to get some advice. >>>>>> PETSC version=3.14.6, program run in serial mode. >>>>>> >>>>>> Regards >>>>>> >>>>>> Yu Cang >>>>> >>>> >>> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hongzhang at anl.gov Mon Nov 8 23:16:25 2021 From: hongzhang at anl.gov (Zhang, Hong) Date: Tue, 9 Nov 2021 05:16:25 +0000 Subject: [petsc-users] Strange behavior of TS after setting hand-coded Jacobian In-Reply-To: References: <0C6ACBF3-F457-4BFD-AD19-8C455444748F@petsc.dev> <9CC15214-4601-4554-808F-C3E96DC3D34A@petsc.dev> <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> Message-ID: <057A6324-1B92-48A4-9B17-D7E4D132208F@anl.gov> On Nov 8, 2021, at 10:59 PM, Zhang, Hong via petsc-users wrote: ? You are essentially solving an index-2 DAE in the form: dx/dt = f(x,y) 0 = g(x) Note that the solvers you have tried, including 'ode15s' and 'ode23' in MATLAB and PETSc TS solvers, are all designed for ODEs or index-1 DAEs, and they cannot be used to solve index-2 DAEs directly. In practice, you need techniques to lower the index of DAEs first. A textbook approach to solve your problem is to differentiate the algebraic equation twice. Taking d/dt for the second equation gives dg/dx dx/dt = 0 ==> dg/dx f(x,y) = 0 At this point, we get an index-1 DAE so we can try to solve it with BEuler or CN in PETSc. The next step enables the application of any ODE solver theoretically. Hong(Mr.) Taking d/dt again gives dg/dx (df/dx f + df/dy dy/dt) = 0 Assuming dg/dx df/dy is not singular, we get dy/dt = -(dg/dx df/dy)^{-1} dg/dx (df/dx f) Now you can solve an ODE system instead of the original DAE. There may exist other approaches for solving this particular problem. But you need to look in the literature. Hong (Mr.) On Nov 7, 2021, at 2:45 AM, ?? > wrote: Thanks for your patient explanation, and sorry for this late reply. I do understand the manipulation skills that you mentioned, but this kind of manually substitution will not change the numerical difficulty of the problem. In fact, before I turn to PETSc, I have tried to form an ODE for this problem already, by utilizing the well-established functions like 'ode15s' and 'ode23' in MATLAB. But it failed to produce physically reasonable results. The key flaw lies in the treatment of the eigenvalue, namely \hat{\Lambda}_r denoted in the PDF file. Only implicit DAE formulation can fully embody its effect. This is why I finally decided to turn to use the DAE functionality in PETSc. I've excerpted some description from a monograph regarding this problem, hope it can clearly demonstrate the numerical requirement for solving it. Kee, R. J., Coltrin, M. E., Glarborg, P., & Zhu, H. (2017). Stagnation Flows. In Chemically Reacting Flow (2 ed., pp. 232). Zhang, Hong > ?2021?11?1??? ??10:21??? You can avoid the zero-diagonal elements in your case. Just remove the boundary points and the \hat{\Lambda}_r from the solution vector Y (eq(17) in your PDE file). You can calculate them separately in your RHS function (when computing the residual vector for 2 <=i<=N-1). It is not necessary to form a DAE. You can actually form an ODE instead. Hong (Mr.) > On Oct 31, 2021, at 5:38 AM, ?? > wrote: > > Thanks for your suggestions and experience sharing. > > But it seems that you have missed my point. This kind of zero-diagonal > element that I described earlier is somehow inevitable. > If interested in this problem, you may refer to the PDF file that was > attached in the previous post. > > Regards > > Yu > > Zhang, Hong > ?2021?10?29??? ??10:05??? >> >> One way to avoid the zero element in Jacobian is to exclude the boundary point from the solution vector. I often do this for Dirichlet boundary conditions since the value at the boundary is given directly and does not need to be taken as a degree of freedom. >> >> Hong (Mr.) >> >> On Oct 28, 2021, at 9:49 PM, ?? > wrote: >> >> Thanks for your careful inspection and thoughtful suggestions. >> >>> finite differencing may put a small non-zero value in that location due to numerical round-off >> >> I think your explanation is reasonable. This numerical round-off may somehow help to avoid this pivot issue. >> >> The structure of my jacobian matrix looks like this (generated by '-mat_view draw'): >> >> Analytically, the first diagonal element of the jacobian is indeed 0, as its corresponding residual function is solely determined from boundary condition of another variable. This seems a little bit wired but is mathematically well-posed. For more description about the background physics, please refer to attached PDF file for more detailed explanation on the discretization and boundary conditions. >> >> Actually, the jacobian matrix is not singular, but I do believe this numerical difficulty is caused by the zero-element in diagonal. >> In this regard, I've performed some trial and test. It seems that several methods have been worked out for this pivot issue: >> a) By setting '-pc_type svd', PETSC does not panic any more with my hand-coded jacobian, and converged solution is obtained. Efficiency is also preserved. >> b) By setting '-pc_type none', converged solution is also obtained, but it takes too many KSP iterations to converge per SNES step (usually hundreds), making the overall solution procedure very slow. >> >> Do you think these methods really solved this kind of pivot issue? Not by chance like the numerical round-off in finite difference previously. >> >> Regards >> >> Yu Cang >> >> Barry Smith > ?2021?10?27??? ??9:43??? >>> >>> >>> You can run with -ksp_error_if_not_converged to get it to stop as soon as a linear solve fails to help track down the exact breaking point. >>> >>>> The problem under consideration contains an eigen-value to be solved, >>>> making the first diagonal element of the jacobian matrix being zero. >>>> From these outputs, it seems that the PC failed to factorize, which is >>>> caused by this 0 diagonal element. But I'm wondering why it works >>>> with jacobian matrix generated by finite-difference? >>> >>> Presumably your "exact" Jacobian puts a zero on the diagonal while the finite differencing may put a small non-zero value in that location due to numerical round-off. In that case even if the factorization succeeds it may produce an inaccurate solution if the value on the diagonal is very small. >>> >>> If your matrix is singular or cannot be factored with LU then you need to use a different solver for the linear system that will be robust to the zero on the diagonal. What is the structure of your Jacobian? (The analytic form). >>> >>> Barry >>> >>> >>>> On Oct 27, 2021, at 1:47 AM, ?? > wrote: >>>> >>>> Thanks for your kind reply. >>>> >>>> Several comparison tests have been performed. Attached are execution >>>> output files. Below are corresponding descriptions. >>>> >>>> good.txt -- Run without hand-coded jacobian, solution converged, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason'; >>>> jac1.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian'; >>>> jac2.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >>>> -ksp_view'; >>>> jac3.txt -- Run with hand-coded jacobian, does not converge, with >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >>>> -ksp_view -ts_max_snes_failures -1 '; >>>> >>>> The problem under consideration contains an eigen-value to be solved, >>>> making the first diagonal element of the jacobian matrix being zero. >>>> From these outputs, it seems that the PC failed to factorize, which is >>>> caused by this 0 diagonal element. But I'm wondering why it works >>>> with jacobian matrix generated by finite-difference? Would employing >>>> DMDA for discretization be helpful? >>>> >>>> Regards >>>> >>>> Yu Cang >>>> >>>> Barry Smith > ?2021?10?25??? ??10:50??? >>>>> >>>>> >>>>> It is definitely unexpected that -snes_test_jacobian verifies the Jacobian as matching but the solve process is completely different. >>>>> >>>>> Please run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian and send all the output >>>>> >>>>> Barry >>>>> >>>>> >>>>>> On Oct 25, 2021, at 9:53 AM, ?? > wrote: >>>>>> >>>>>> I'm using TS to solve a set of DAE, which originates from a >>>>>> one-dimensional problem. The grid points are uniformly distributed. >>>>>> For simplicity, the DMDA is not employed for discretization. >>>>>> >>>>>> At first, only the residual function is prescribed through >>>>>> 'TSSetIFunction', and PETSC produces converged results. However, after >>>>>> providing hand-coded Jacobian through 'TSSetIJacobian', the internal >>>>>> SNES object fails (residual norm does not change), and TS reports >>>>>> 'DIVERGED_STEP_REJECTED'. >>>>>> >>>>>> I have tried to add the option '-snes_test_jacobian' to see if the >>>>>> hand-coded jacobian is somewhere wrong, but it shows '||J - >>>>>> Jfd||_F/||J||_F = 1.07488e-10, ||J - Jfd||_F = 2.14458e-07', >>>>>> indicating that the hand-coded jacobian is correct. >>>>>> >>>>>> Then, I added a monitor for the internal SNES object through >>>>>> 'SNESMonitorSet', in which the solution vector will be displayed at >>>>>> each iteration. It is interesting to find that, if the jacobian is not >>>>>> provided, meaning finite-difference is utilized for jacobian >>>>>> evaluation internally, the solution vector converges to steady >>>>>> solution and the SNES residual norm is reduced continuously. However, >>>>>> it turns out that, as long as the jacobian is provided, the solution >>>>>> vector will NEVER get changed! So the solution procedure stucked! >>>>>> >>>>>> This is quite strange! Hope to get some advice. >>>>>> PETSC version=3.14.6, program run in serial mode. >>>>>> >>>>>> Regards >>>>>> >>>>>> Yu Cang >>>>> >>>> >>> >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 9 04:41:33 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Nov 2021 05:41:33 -0500 Subject: [petsc-users] Problem with using make In-Reply-To: References: Message-ID: On Mon, Nov 8, 2021 at 9:57 PM Yaswanth Narayana wrote: > Hello > > My name is Yaswanth Narayana. I am using PETSc as part of my course HPC1. > When trying to use make I am getting the following error. What should I do? > > makefile:8: /lib/petsc/conf/rules: No such file or directory > make: *** No rule to make target 'PETSC_DIR/lib/petsc/conf/rules'. Stop. > You do not have PETSC_DIR defined in your environment. You can check this using echo $PETSC_DIR It should print a directory name. Thanks, Prof. Knepley > I have tried changing the directory name and changed to the directory > containing files 'rules' and 'variables' but still receiving the same > error. Please help in this regard. > > Thank you > > Yours sincerely > Yaswanth > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 9 04:46:42 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Nov 2021 05:46:42 -0500 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> References: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> Message-ID: On Tue, Nov 9, 2021 at 4:24 AM Nicolas Barnafi wrote: > > I am not adept enough at FEniCS to look at the code and debug. However, > I would make a very small problem for 2 processes, say two cells on each, > and then print out is_0 and is_1. That should tell you if you have the > right > unknowns in each block. If so, then it seems like something is not correct > in > your assembly routine, which is probably better debugged on the FEniCS > list. > However, if they are not right, we can help get them right. > > THanks, > > Matt > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > Thank Matthew, I had been testing the indexes and they looked fine. In > fact: > > Rank-0, is_0: [0 1 2] [633 634 635] > Rank-0, is_1 [636 637 638] [729 730 731] > Rank-1, is_0: [732 733 734] [1368 1369 1370] > Rank-1, is_1 [1371 1372 1373] [1464 1465 1466] > > where is_1 should be much smaller, and it is indeed the case. > I am now completely confused. The way PCFIELDSPLIT works, when doing Schur complements, is to take in two ISes, which can be thought of as lists of vector rows. These two ISes must partition [0, N) if N is the global size of the problem. So 1) The numbers you have above do not appear to be a partition 2) The problem appears to be huge. Use a problem with 2-4 to debug. Thanks, Matt > Thanks! > NB > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nabw91 at gmail.com Tue Nov 9 05:17:59 2021 From: nabw91 at gmail.com (Nicolas Barnafi) Date: Tue, 9 Nov 2021 12:17:59 +0100 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: References: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> Message-ID: <4c24a018-fc8c-0a02-5e8b-156a248ba270@gmail.com> Whoops, I think the notation wasn't very good, sorry. This should have been my previous message: Rank-0, is_0: [0 1 2] ... [633 634 635] Rank-0, is_1 [636 637 638] ... [729 730 731] Rank-1, is_0: [732 733 734] ... [1368 1369 1370] Rank-1, is_1 [1371 1372 1373] ... [1464 1465 1466] where I am printing the three first and last elements of each IS object, so in fact the first processor has the range [0,635] (is0) + [636,731] (is1) and the second one has [732, 1370] (is0) + [1371, 1466] (is1). This is actually the smallest (3D) problem I have access to without changing anything, as I have hard-coded some elements (i.e. boundary conditions). It still runs super quickly, so it is not much of a problem to debug it as it is. Thanks you, NB > > I am now completely confused. The way PCFIELDSPLIT works, when doing > Schur complements, is to take > in two ISes, which can be thought of as lists of vector rows. These > two ISes must partition [0, N) if N is the > global size of the problem. So > > ? 1) The numbers you have above do not appear to be a partition > > ? 2) The problem appears to be huge. Use a problem with 2-4 to debug. > > ? Thanks, > > ? ? ?Matt > > Thanks! > NB > > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 9 05:25:53 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Nov 2021 06:25:53 -0500 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: <4c24a018-fc8c-0a02-5e8b-156a248ba270@gmail.com> References: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> <4c24a018-fc8c-0a02-5e8b-156a248ba270@gmail.com> Message-ID: On Tue, Nov 9, 2021 at 6:18 AM Nicolas Barnafi wrote: > Whoops, I think the notation wasn't very good, sorry. This should have > been my previous message: > > Rank-0, is_0: [0 1 2] ... [633 634 635] > Rank-0, is_1 [636 637 638] ... [729 730 731] > Rank-1, is_0: [732 733 734] ... [1368 1369 1370] > Rank-1, is_1 [1371 1372 1373] ... [1464 1465 1466] > > where I am printing the three first and last elements of each IS object, > so in fact the first processor has the range [0,635] (is0) + [636,731] > (is1) and the second one has [732, 1370] (is0) + [1371, 1466] (is1). This > is actually the smallest (3D) problem I have access to without changing > anything, as I have hard-coded some elements (i.e. boundary conditions). It > still runs super quickly, so it is not much of a problem to debug it as it > is. > The next step would be to verify that the operators you get in those slots are correct in parallel. I normally just print out the matrix for a small problem. This is the only way I know to debug assembly. To check the solver, you set it to an exact solver and see that it takes 1 iterate. That means using LU as the subsolver for the first block, using the full Schur complement, and solving the Schur system with a residual tolerance of something like 1e-10. Thanks, Matt > Thanks you, > NB > > > I am now completely confused. The way PCFIELDSPLIT works, when doing Schur > complements, is to take > in two ISes, which can be thought of as lists of vector rows. These two > ISes must partition [0, N) if N is the > global size of the problem. So > > 1) The numbers you have above do not appear to be a partition > > 2) The problem appears to be huge. Use a problem with 2-4 to debug. > > Thanks, > > Matt > > >> Thanks! >> NB >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nabw91 at gmail.com Tue Nov 9 05:52:45 2021 From: nabw91 at gmail.com (Nicolas Barnafi) Date: Tue, 9 Nov 2021 12:52:45 +0100 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: References: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> <4c24a018-fc8c-0a02-5e8b-156a248ba270@gmail.com> Message-ID: <3b5cd7fe-3203-b37e-53b6-900a7d1b64f5@gmail.com> Thanks for the help Matthew. > > The next?step would be to verify that the operators you get in those > slots are correct in parallel. I normally just print out the matrix > for a small problem. This is the only way I know to debug assembly. > > To check the solver, you set it to an exact solver and see that it > takes 1 iterate. That means using LU as the subsolver > for the first block, using the full Schur complement, and solving the > Schur system with a residual tolerance of something like 1e-10. > There is certainly something wrong in the nested matrix, as this is the output in serial (LU + full Schur LU): ? 0 SNES Function norm 3.240309155660e+03 ??? 0 KSP Residual norm 3.240309155660e+03 ??? 1 KSP Residual norm 3.219893365194e-08 ? 1 SNES Function norm 3.999872813081e+03 ??? 0 KSP Residual norm 1.214145680994e+04 ??? 1 KSP Residual norm 2.742817643080e-04 ? 2 SNES Function norm 3.663148395735e+01 ??? 0 KSP Residual norm 3.980103473350e+03 ??? 1 KSP Residual norm 1.814754406828e-05 ? 3 SNES Function norm 2.136252100504e+01 ??? 0 KSP Residual norm 6.970609974245e+01 ??? 1 KSP Residual norm 1.982821753925e-05 ? 4 SNES Function norm 1.059694009734e-02 ??? 0 KSP Residual norm 2.136131725219e+01 ??? 1 KSP Residual norm 4.545878835451e-06 ??? 2 KSP Residual norm 8.841427426699e-08 ??? 3 KSP Residual norm 3.305898584949e-09 ? 5 SNES Function norm 3.729120062850e-03 ??? 0 KSP Residual norm 3.846880232871e-03 ??? 1 KSP Residual norm 1.740268533385e-06 ??? 2 KSP Residual norm 3.013108797112e-08 ??? 3 KSP Residual norm 1.105602265959e-09 ? 6 SNES Function norm 1.117697448247e-09 whereas in parallel it is this (two processes): ? 0 SNES Function norm 3.240309155660e+03 ??? 0 KSP Residual norm 3.240309155660e+03 ??? 1 KSP Residual norm 2.158162432432e-01 ??? 2 KSP Residual norm 2.156121452805e-01 ??? 3 KSP Residual norm 2.148639482595e-01 ??? 4 KSP Residual norm 2.148248193325e-01 ??? 5 KSP Residual norm 1.305140166188e-03 ? Linear solve converged due to CONVERGED_RTOL iterations 5 ? 1 SNES Function norm 4.774611203034e+42 Best, NB -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.stone at opengosim.com Tue Nov 9 05:55:34 2021 From: daniel.stone at opengosim.com (Daniel Stone) Date: Tue, 9 Nov 2021 11:55:34 +0000 Subject: [petsc-users] Configure of 3.16.1 failing on mpich/yaksa In-Reply-To: <801382a1-a618-dcf9-e7da-a868343344d@mcs.anl.gov> References: <801382a1-a618-dcf9-e7da-a868343344d@mcs.anl.gov> Message-ID: Just for interest, I found out that the decision to use cuda comes from Yaksa checking for cuda_runtime.h and such files as part of its configure process - so it's not a decision made by the petsc or even mpich scripts. On the machine with cuda 11 installed manually, it can't find the header files, understandably as it hasn't been told where to look (with_cuda_dir=.... etc not set). On the machine where cuda 10 is installed via apt-get, it can find them - presumably because it looks in /usr/include/ or similar by default - some standard place where apt-get would put the headers. Thanks for the advice on various workarounds - I had forgotten that one could do stuff like --download-mpich= https://www.mpich.org/static/downloads/3.3.2/mpich-3.3.2.tar.gz which would probably be the cleanest option. On Mon, Nov 8, 2021 at 7:36 PM Satish Balay wrote: > This looks like a bug report to mpich. > > You should be able to reproduce this without PETSc - by directly building > MPICH. > > And then report to MPICH developers. > > Wrt petsc - you could try --download-openmpi [instead of --download-mpich] > and see if that works better. > > Yeah cuda obtained from nvida and cuda repackaged by ubuntu have subtle > differences that can cause cuda failures. > > As you say - mpich might have a configure option to disable this. If you > are able to find this option - you can use it via petsc configure with: > > --download-mpich-configure-arguments=string > > Satish > > On Mon, 8 Nov 2021, Daniel Stone wrote: > > > Hello all, > > > > I've been having some configure failures trying to configure petsc, on > > Ubuntu 20, when > > downloading mpich. > > > > > > This seems to be related to the use of > > "#!/bin/sh" > > found in the script > > mpich-3.4.2/modules/yaksa/src/backend/cuda/cudalt.sh > > > > /bin/sh in Ubuntu20 is dash, not bash, and line 35 of the script is: > > CMD="${@:2} -Xcompiler -fPIC -o $PIC_FILEPATH" > > which is apparently not valid dash syntax. I see "bad substitution" > errors > > when > > trying to run this script in isolation, which can be fixed by replacing > the > > top line with > > > > "#!/bin/bash" > > > > The petsc config log points to this line in this script: > > > > > ------------------------------------------------------------------------------------ > > > > make[2]: Entering directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hindexed__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_hvector_contig__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hvector__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_resized__Bool.lo > > NVCC src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_blkhindx__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_hindexed__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_contig__Bool.lo > > NVCC > src/backend/cuda/pup/yaksuri_cudai_pup_blkhindx_resized__Bool.lo > > make[2]: Leaving directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2/modules/yaksa' > > make[1]: Leaving directory > > '/home/david/petsc/petsc_opt/externalpackages/mpich-3.4.2'/usr/bin/ar: > `u' > > modifier ignored since `D' is the default (see `U') > > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > > /usr/bin/ar: `u' modifier ignored since `D' is the default (see `U') > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > ./src/backend/cuda/cudalt.sh: 35: Bad substitution > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector__Bool.lo] Error 2 > > make[2]: *** Waiting for unfinished jobs.... > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_hvector__Bool.lo] Error 2 > > make[2]: *** [Makefile:8697: > > src/backend/cuda/pup/yaksuri_cudai_pup_hvector_blkhindx__Bool.lo] Error 2 > > > > > ---------------------------------------------------------------------------- > > > > What is interesting is the choice made by the config script here to make > > yaksa "cuda-aware", which I do not > > understand how to control. By this I mean - the use of NVCC, the use of > > files with "cudai" in the name, > > and the running of the cudalt.sh script. > > > > This is especially odd given that on another machine, also with Ubuntu20, > > none of this occurs, despite > > using the exact same configure instructions: > > > > --------------------------------------------------- > > CC src/backend/seq/pup/yaksuri_seqi_pup_blkhindx_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hvector__Bool.lo > > CC > src/backend/seq/pup/yaksuri_seqi_pup_hindexed_blkhindx__Bool.lo > > CC > src/backend/seq/pup/yaksuri_seqi_pup_hindexed_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_hindexed_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hvector__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_blkhindx__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_contig_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hvector__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_blkhindx__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_hindexed__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_contig__Bool.lo > > CC src/backend/seq/pup/yaksuri_seqi_pup_resized_resized__Bool.lo > > > > > > ------------------------------------------------------- > > > > On both machines petsc locates a working nvcc without trouble. One > > difference is > > that on the good machine, cuda 11 is installed, built from source, while > on > > the bad machine > > cuda 10 is installed, installed via apt-get. On the good machine, running > > the cudalt.sh script in > > isolation results in the same bad substitution error. > > > > Can someone help me understand where the difference of behaviour might > come > > from, w.r.t. > > cuda? Adding the --with-cuda=0 flag on the bad machine made no > difference. > > Is there any way > > of communicating to mpich/yaksuri that I don't want whatever features > that > > involve the cudalt.sh > > script being run? > > > > The configure command I use is: > > > > ./configure --download-mpich=yes --download-hdf5=yes > > --download-fblaslapack=yes --download-metis=yes --download-cmake=yes > > --download-ptscotch=yes --download-hypre=yes --with-debugging=0 > > COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 > > -download-hdf5-fortran-bindings=yes --download-sowing > > > > > > > > Thanks, > > > > Daniel > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 9 06:01:22 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Nov 2021 07:01:22 -0500 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: <3b5cd7fe-3203-b37e-53b6-900a7d1b64f5@gmail.com> References: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> <4c24a018-fc8c-0a02-5e8b-156a248ba270@gmail.com> <3b5cd7fe-3203-b37e-53b6-900a7d1b64f5@gmail.com> Message-ID: On Tue, Nov 9, 2021 at 6:52 AM Nicolas Barnafi wrote: > Thanks for the help Matthew. > > > The next step would be to verify that the operators you get in those slots > are correct in parallel. I normally just print out the matrix > for a small problem. This is the only way I know to debug assembly. > > To check the solver, you set it to an exact solver and see that it takes 1 > iterate. That means using LU as the subsolver > for the first block, using the full Schur complement, and solving the > Schur system with a residual tolerance of something like 1e-10. > > > There is certainly something wrong in the nested matrix, as this is the > output in serial (LU + full Schur LU): > Yes, this is why I recommend going to just 2 or 4 cells, so that you can see explicitly what is wrong with the matrix. Thanks, Matt > 0 SNES Function norm 3.240309155660e+03 > 0 KSP Residual norm 3.240309155660e+03 > 1 KSP Residual norm 3.219893365194e-08 > 1 SNES Function norm 3.999872813081e+03 > 0 KSP Residual norm 1.214145680994e+04 > 1 KSP Residual norm 2.742817643080e-04 > 2 SNES Function norm 3.663148395735e+01 > 0 KSP Residual norm 3.980103473350e+03 > 1 KSP Residual norm 1.814754406828e-05 > 3 SNES Function norm 2.136252100504e+01 > 0 KSP Residual norm 6.970609974245e+01 > 1 KSP Residual norm 1.982821753925e-05 > 4 SNES Function norm 1.059694009734e-02 > 0 KSP Residual norm 2.136131725219e+01 > 1 KSP Residual norm 4.545878835451e-06 > 2 KSP Residual norm 8.841427426699e-08 > 3 KSP Residual norm 3.305898584949e-09 > 5 SNES Function norm 3.729120062850e-03 > 0 KSP Residual norm 3.846880232871e-03 > 1 KSP Residual norm 1.740268533385e-06 > 2 KSP Residual norm 3.013108797112e-08 > 3 KSP Residual norm 1.105602265959e-09 > 6 SNES Function norm 1.117697448247e-09 > > whereas in parallel it is this (two processes): > > 0 SNES Function norm 3.240309155660e+03 > 0 KSP Residual norm 3.240309155660e+03 > 1 KSP Residual norm 2.158162432432e-01 > 2 KSP Residual norm 2.156121452805e-01 > 3 KSP Residual norm 2.148639482595e-01 > 4 KSP Residual norm 2.148248193325e-01 > 5 KSP Residual norm 1.305140166188e-03 > Linear solve converged due to CONVERGED_RTOL iterations 5 > 1 SNES Function norm 4.774611203034e+42 > > > Best, > NB > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Nov 9 07:14:24 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Nov 2021 08:14:24 -0500 Subject: [petsc-users] Problem with using make In-Reply-To: References: <113248C5-425E-4140-8255-51304A5F8A52@petsc.dev> Message-ID: <5A67B0E6-5BCA-4D43-896A-F754EEF29862@petsc.dev> Please send configure.log and make.log to petsc-maint at mcs.anl.gov as well as all the output from trying to compiled ex5.c > On Nov 9, 2021, at 12:33 AM, Yaswanth Narayana wrote: > > I reinstalled PETSc but still getting the same error message. > > For another problem, I tried compiling ex5.c in tutorials(in snes) but got following error > > ex5.c:50:10: fatal error: petscdm.h: No such file or directory > 50 | #include > | ^~~~~~~~~~~ > compilation terminated. > > May I know what went wrong here? > > Thank you > > Yours sincerely > Yaswanth > > From: Barry Smith > Sent: Tuesday, November 9, 2021 8:46 AM > To: Yaswanth Narayana > Cc: petsc-users at mcs.anl.gov > Subject: Re: [petsc-users] Problem with using make > > > It seems you missed $ in front of $PETSC_DIR somewhere along the line. I suggest deleting the PETSc directory completely, recloning PETSc and follow the instructions at https://petsc.org/release/install/ very careful to insure there are no typos or missing symbols. > > Barry > > >> On Nov 8, 2021, at 9:51 PM, Yaswanth Narayana > wrote: >> >> Hello >> >> My name is Yaswanth Narayana. I am using PETSc as part of my course HPC1. When trying to use make I am getting the following error. What should I do? >> >> makefile:8: /lib/petsc/conf/rules: No such file or directory >> make: *** No rule to make target 'PETSC_DIR/lib/petsc/conf/rules'. Stop. >> >> I have tried changing the directory name and changed to the directory containing files 'rules' and 'variables' but still receiving the same error. Please help in this regard. >> >> Thank you >> >> Yours sincerely >> Yaswanth -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.barnafi at e.email Tue Nov 9 03:24:43 2021 From: nicolas.barnafi at e.email (Nicolas Barnafi) Date: Tue, 9 Nov 2021 10:24:43 +0100 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: References: Message-ID: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> > I am not adept enough at FEniCS to look at the code and debug. However, > I would make a very small problem for 2 processes, say two cells on each, > and then print out is_0 and is_1. That should tell you if you have the > right > unknowns in each block. If so, then it seems like something is not > correct in > your assembly routine, which is probably better debugged on the FEniCS > list. > However, if they are not right, we can help get them right. > > ? THanks, > > ? ? Matt > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > Thank Matthew, I had been testing the indexes and they looked fine. In fact: Rank-0, is_0: [0 1 2] [633 634 635] Rank-0, is_1 [636 637 638] [729 730 731] Rank-1, is_0: [732 733 734] [1368 1369 1370] Rank-1, is_1 [1371 1372 1373] [1464 1465 1466] where is_1 should be much smaller, and it is indeed the case. Thanks! NB -------------- next part -------------- An HTML attachment was scrubbed... URL: From maguprasaad at iisc.ac.in Tue Nov 9 04:15:47 2021 From: maguprasaad at iisc.ac.in (Magu Raam Prasaad R) Date: Tue, 9 Nov 2021 10:15:47 +0000 Subject: [petsc-users] Error loading vectors from files : Too many open files. Message-ID: Hello all, I am getting this error message while loading vectors from files. I have attached the code to reproduce the error message. Steps to compile and run the code can be found in monople/README Machine type: WorkStation OS version : Ubuntu 17.04 PETSC version: 3.14.4 Compiler : GNU C MPI: mpich Petsc Components: Vec [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Unable to open file [0]PETSC ERROR: Cannot open file p-00502.dat for READ: ??H?E???? [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.14.4, unknown [0]PETSC ERROR: ./kirchhoff on a linux-gnu-c-debug named galois by ratnesh Tue Nov 9 15:05:43 2021 [0]PETSC ERROR: Configure options --download-triangle --download-ctetgen --download-chaco --download-metis --download-parmetis --download-hdf5 [0]PETSC ERROR: #1 PetscBinaryOpen() line 505 in /home/ratnesh/petsc/src/sys/fileio/sysio.c [0]PETSC ERROR: #2 PetscViewerFileSetUp_BinarySTDIO() line 1457 in /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c [0]PETSC ERROR: #3 PetscViewerSetUp_Binary() line 1512 in /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c [0]PETSC ERROR: #4 PetscViewerSetUp() line 328 in /home/ratnesh/petsc/src/sys/classes/viewer/interface/view.c [0]PETSC ERROR: #5 VecLoad_Binary() line 73 in /home/ratnesh/petsc/src/vec/vec/utils/vecio.c [0]PETSC ERROR: #6 VecLoad_Default() line 279 in /home/ratnesh/petsc/src/vec/vec/utils/vecio.c [0]PETSC ERROR: #7 VecLoad() line 950 in /home/ratnesh/petsc/src/vec/vec/interface/vector.c [0]PETSC ERROR: #8 ComputePressureAtEmissionTime() line 239 in /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c [0]PETSC ERROR: #9 main() line 425 in /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -vecload_block_size 1 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- Sorry! You were supposed to get help about: mpi-abort But I couldn't open the help file: /usr/share/openmpi/help-mpi-api.txt: Too many open files. Sorry! Thank you. Regards Magu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: monopole.zip Type: application/zip Size: 20830 bytes Desc: monopole.zip URL: From bhargav.subramanya at kaust.edu.sa Tue Nov 9 08:47:52 2021 From: bhargav.subramanya at kaust.edu.sa (Bhargav Subramanya) Date: Tue, 9 Nov 2021 17:47:52 +0300 Subject: [petsc-users] dmplex extrude in parallel Message-ID: Dear All, I want to generate a prismatic mesh from a base spherical surface mesh. I first refine the base icosahedron sequential mesh to some extent, distribute the mesh, continue the refinement in parallel, project the mesh to a unit sphere, and then finally extrude the mesh. The following is the code I am using. As the extrusion is performed in parallel, the extruded mesh seems to be broken as shown in the attached figure. May I know how to get an intact extruded mesh in parallel? Also, is it possible to make mesh refinement respect the spherical surface geometry, without having to project it using a function as shown below? Thanks, Bhargav /* refine the sequential mesh first */ for (int r = 0; r < numRefinePre; ++r) { DM rdm = NULL; DMRefine(dm, MPI_COMM_WORLD, &rdm); DMDestroy(&dm); dm = rdm; } /* project to a unit sphere */ ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); /* create and distribute DM */ ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); if (dmDist) { DMDestroy(&dm); dm = dmDist; } /* refine the mesh in parallel */ for (int r = 0; r < numRefine; ++r) { DM rdm = NULL; DMRefine(dm, MPI_COMM_WORLD, &rdm); DMDestroy(&dm); dm = rdm; } /* project to a unit sphere */ ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); ierr = DMPlexExtrude(dm, numLayers-1, radialThickness, PETSC_FALSE, NULL, PETSC_TRUE, &dm3D);CHKERRQ(ierr); if (dm3D) { DMDestroy(&dm); dm = dm3D; } -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: extruded-mesh.pdf Type: application/pdf Size: 15372160 bytes Desc: not available URL: From mfadams at lbl.gov Tue Nov 9 10:38:55 2021 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 9 Nov 2021 11:38:55 -0500 Subject: [petsc-users] Error loading vectors from files : Too many open files. In-Reply-To: References: Message-ID: On Tue, Nov 9, 2021 at 9:09 AM Magu Raam Prasaad R via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello all, > > I am getting this error message while loading vectors from files. I have > attached the code to reproduce the error message. > > Steps to compile and run the code can be found in monople/README > Machine type: WorkStation > OS version : Ubuntu 17.04 > PETSC version: 3.14.4 > Compiler : GNU C > MPI: mpich > Petsc Components: Vec > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Unable to open file > [0]PETSC ERROR: Cannot open file p-00502.dat for READ: ??H?E?? ?? > I don't see this .dat file. Maybe try the full path name or add the file to the working directory. > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.14.4, unknown > [0]PETSC ERROR: ./kirchhoff on a linux-gnu-c-debug named galois by ratnesh > Tue Nov 9 15:05:43 2021 > [0]PETSC ERROR: Configure options --download-triangle --download-ctetgen > --download-chaco --download-metis --download-parmetis --download-hdf5 > [0]PETSC ERROR: #1 PetscBinaryOpen() line 505 in > /home/ratnesh/petsc/src/sys/fileio/sysio.c > [0]PETSC ERROR: #2 PetscViewerFileSetUp_BinarySTDIO() line 1457 in > /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c > [0]PETSC ERROR: #3 PetscViewerSetUp_Binary() line 1512 in > /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c > [0]PETSC ERROR: #4 PetscViewerSetUp() line 328 in > /home/ratnesh/petsc/src/sys/classes/viewer/interface/view.c > [0]PETSC ERROR: #5 VecLoad_Binary() line 73 in > /home/ratnesh/petsc/src/vec/vec/utils/vecio.c > [0]PETSC ERROR: #6 VecLoad_Default() line 279 in > /home/ratnesh/petsc/src/vec/vec/utils/vecio.c > [0]PETSC ERROR: #7 VecLoad() line 950 in > /home/ratnesh/petsc/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #8 ComputePressureAtEmissionTime() line 239 in > /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c > [0]PETSC ERROR: #9 main() line 425 in > /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -vecload_block_size 1 > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > Sorry! You were supposed to get help about: > mpi-abort > But I couldn't open the help file: > /usr/share/openmpi/help-mpi-api.txt: Too many open files. Sorry! > > Thank you. > > Regards > Magu > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nabw91 at gmail.com Tue Nov 9 11:27:00 2021 From: nabw91 at gmail.com (barnafi) Date: Tue, 09 Nov 2021 18:27:00 +0100 Subject: [petsc-users] Fwd: SNES with Fieldsplit + MatNest in parallel In-Reply-To: References: <3cc2b606-ab45-1028-d0a7-e3af4f483f41@e.email> <4c24a018-fc8c-0a02-5e8b-156a248ba270@gmail.com> <3b5cd7fe-3203-b37e-53b6-900a7d1b64f5@gmail.com> Message-ID: <05FB2R.33UIXN6094FX1@gmail.com> I have checked the matrix in a very small problem and they are the same, so the problem is definitely somewhere in the ordering/ghost interface from fenics, so PETSc is doing everything fine. Thanks for your help, I'm moving to the fenics forum. Best, NB On Tue, Nov 9, 2021 at 07:01, Matthew Knepley wrote: > On Tue, Nov 9, 2021 at 6:52 AM Nicolas Barnafi > wrote: >> Thanks for the help Matthew. >>> >>> The next step would be to verify that the operators you get in >>> those slots are correct in parallel. I normally just print out the >>> matrix >>> for a small problem. This is the only way I know to debug assembly. >>> >>> To check the solver, you set it to an exact solver and see that it >>> takes 1 iterate. That means using LU as the subsolver >>> for the first block, using the full Schur complement, and solving >>> the Schur system with a residual tolerance of something like 1e-10. >>> >> >> There is certainly something wrong in the nested matrix, as this is >> the output in serial (LU + full Schur LU): > > Yes, this is why I recommend going to just 2 or 4 cells, so that you > can see explicitly what is wrong with the matrix. > > Thanks, > > Matt > >> 0 SNES Function norm 3.240309155660e+03 >> 0 KSP Residual norm 3.240309155660e+03 >> 1 KSP Residual norm 3.219893365194e-08 >> 1 SNES Function norm 3.999872813081e+03 >> 0 KSP Residual norm 1.214145680994e+04 >> 1 KSP Residual norm 2.742817643080e-04 >> 2 SNES Function norm 3.663148395735e+01 >> 0 KSP Residual norm 3.980103473350e+03 >> 1 KSP Residual norm 1.814754406828e-05 >> 3 SNES Function norm 2.136252100504e+01 >> 0 KSP Residual norm 6.970609974245e+01 >> 1 KSP Residual norm 1.982821753925e-05 >> 4 SNES Function norm 1.059694009734e-02 >> 0 KSP Residual norm 2.136131725219e+01 >> 1 KSP Residual norm 4.545878835451e-06 >> 2 KSP Residual norm 8.841427426699e-08 >> 3 KSP Residual norm 3.305898584949e-09 >> 5 SNES Function norm 3.729120062850e-03 >> 0 KSP Residual norm 3.846880232871e-03 >> 1 KSP Residual norm 1.740268533385e-06 >> 2 KSP Residual norm 3.013108797112e-08 >> 3 KSP Residual norm 1.105602265959e-09 >> 6 SNES Function norm 1.117697448247e-09 >> >> whereas in parallel it is this (two processes): >> >> 0 SNES Function norm 3.240309155660e+03 >> 0 KSP Residual norm 3.240309155660e+03 >> 1 KSP Residual norm 2.158162432432e-01 >> 2 KSP Residual norm 2.156121452805e-01 >> 3 KSP Residual norm 2.148639482595e-01 >> 4 KSP Residual norm 2.148248193325e-01 >> 5 KSP Residual norm 1.305140166188e-03 >> Linear solve converged due to CONVERGED_RTOL iterations 5 >> 1 SNES Function norm 4.774611203034e+42 >> >> >> Best, >> NB > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From mfadams at lbl.gov Tue Nov 9 13:08:24 2021 From: mfadams at lbl.gov (Mark Adams) Date: Tue, 9 Nov 2021 14:08:24 -0500 Subject: [petsc-users] Error loading vectors from files : Too many open files. In-Reply-To: References: Message-ID: Petsc just failed to open this file. Is this .dat file a PETSc binary file? Try to run the PETSc example in https://petsc.org/release/docs/manualpages/Sys/PetscBinaryOpen.html On Tue, Nov 9, 2021 at 9:09 AM Magu Raam Prasaad R via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello all, > > I am getting this error message while loading vectors from files. I have > attached the code to reproduce the error message. > > Steps to compile and run the code can be found in monople/README > Machine type: WorkStation > OS version : Ubuntu 17.04 > PETSC version: 3.14.4 > Compiler : GNU C > MPI: mpich > Petsc Components: Vec > > > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Unable to open file > [0]PETSC ERROR: Cannot open file p-00502.dat for READ: ??H?E?? ?? > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.14.4, unknown > [0]PETSC ERROR: ./kirchhoff on a linux-gnu-c-debug named galois by ratnesh > Tue Nov 9 15:05:43 2021 > [0]PETSC ERROR: Configure options --download-triangle --download-ctetgen > --download-chaco --download-metis --download-parmetis --download-hdf5 > [0]PETSC ERROR: #1 PetscBinaryOpen() line 505 in > /home/ratnesh/petsc/src/sys/fileio/sysio.c > [0]PETSC ERROR: #2 PetscViewerFileSetUp_BinarySTDIO() line 1457 in > /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c > [0]PETSC ERROR: #3 PetscViewerSetUp_Binary() line 1512 in > /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c > [0]PETSC ERROR: #4 PetscViewerSetUp() line 328 in > /home/ratnesh/petsc/src/sys/classes/viewer/interface/view.c > [0]PETSC ERROR: #5 VecLoad_Binary() line 73 in > /home/ratnesh/petsc/src/vec/vec/utils/vecio.c > [0]PETSC ERROR: #6 VecLoad_Default() line 279 in > /home/ratnesh/petsc/src/vec/vec/utils/vecio.c > [0]PETSC ERROR: #7 VecLoad() line 950 in > /home/ratnesh/petsc/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #8 ComputePressureAtEmissionTime() line 239 in > /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c > [0]PETSC ERROR: #9 main() line 425 in > /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -vecload_block_size 1 > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > -------------------------------------------------------------------------- > Sorry! You were supposed to get help about: > mpi-abort > But I couldn't open the help file: > /usr/share/openmpi/help-mpi-api.txt: Too many open files. Sorry! > > Thank you. > > Regards > Magu > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Nov 9 14:31:52 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Nov 2021 15:31:52 -0500 Subject: [petsc-users] Error loading vectors from files : Too many open files. In-Reply-To: References: Message-ID: <8C80B567-7914-4327-BFFC-876334AE2DB6@petsc.dev> The code is likely creating many viewers each for its own file and not destroying the viewers resulting all the files remaining open. If you are explicitly creating viewers with PetscViewerBinaryOpen() make sure to destroy it with PetscViewerDestroy() after loading from it. Barry > On Nov 9, 2021, at 5:15 AM, Magu Raam Prasaad R via petsc-users wrote: > > Hello all, > > I am getting this error message while loading vectors from files. I have attached the code to reproduce the error message. > > Steps to compile and run the code can be found in monople/README > Machine type: WorkStation > OS version : Ubuntu 17.04 > PETSC version: 3.14.4 > Compiler : GNU C > MPI: mpich > Petsc Components: Vec > > > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > [0]PETSC ERROR: Unable to open file > [0]PETSC ERROR: Cannot open file p-00502.dat for READ: ??H?E???? > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.14.4, unknown > [0]PETSC ERROR: ./kirchhoff on a linux-gnu-c-debug named galois by ratnesh Tue Nov 9 15:05:43 2021 > [0]PETSC ERROR: Configure options --download-triangle --download-ctetgen --download-chaco --download-metis --download-parmetis --download-hdf5 > [0]PETSC ERROR: #1 PetscBinaryOpen() line 505 in /home/ratnesh/petsc/src/sys/fileio/sysio.c > [0]PETSC ERROR: #2 PetscViewerFileSetUp_BinarySTDIO() line 1457 in /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c > [0]PETSC ERROR: #3 PetscViewerSetUp_Binary() line 1512 in /home/ratnesh/petsc/src/sys/classes/viewer/impls/binary/binv.c > [0]PETSC ERROR: #4 PetscViewerSetUp() line 328 in /home/ratnesh/petsc/src/sys/classes/viewer/interface/view.c > [0]PETSC ERROR: #5 VecLoad_Binary() line 73 in /home/ratnesh/petsc/src/vec/vec/utils/vecio.c > [0]PETSC ERROR: #6 VecLoad_Default() line 279 in /home/ratnesh/petsc/src/vec/vec/utils/vecio.c > [0]PETSC ERROR: #7 VecLoad() line 950 in /home/ratnesh/petsc/src/vec/vec/interface/vector.c > [0]PETSC ERROR: #8 ComputePressureAtEmissionTime() line 239 in /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c > [0]PETSC ERROR: #9 main() line 425 in /home/ratnesh/Desktop/kirchhoff/monopole/kirchhoff/kirchhoff.c > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -vecload_block_size 1 > [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov ---------- > -------------------------------------------------------------------------- > Sorry! You were supposed to get help about: > mpi-abort > But I couldn't open the help file: > /usr/share/openmpi/help-mpi-api.txt: Too many open files. Sorry! > > Thank you. > > Regards > Magu > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Tue Nov 9 15:56:28 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 9 Nov 2021 16:56:28 -0500 Subject: [petsc-users] dmplex extrude in parallel In-Reply-To: References: Message-ID: On Tue, Nov 9, 2021 at 9:54 AM Bhargav Subramanya < bhargav.subramanya at kaust.edu.sa> wrote: > Dear All, > > I want to generate a prismatic mesh from a base spherical surface mesh. I > first refine the base icosahedron sequential mesh to some extent, > distribute the mesh, continue the refinement in parallel, project the mesh > to a unit sphere, and then finally extrude the mesh. The following is the > code I am using. As the extrusion is performed in parallel, the extruded > mesh seems to be broken as shown in the attached figure. May I know how to > get an intact extruded mesh in parallel? Also, is it possible to make mesh > refinement respect the spherical surface geometry, without having to > project it using a function as shown below? > I think you can do most of what you want from the command line. Here is a simple code: #include int main(int argc, char **argv) { DM dm; PetscErrorCode ierr; ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); ierr = DMSetFromOptions(dm);CHKERRQ(ierr); ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); ierr = DMDestroy(&dm);CHKERRQ(ierr); ierr = PetscFinalize(); return ierr; } that I run using mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 -dm_extrude 5 -dm_plex_transform_extrude_thickness 0.1 -dm_distribute -dm_view hdf5:mesh.h5 and get the attached picture. I am refining before distribution only here, since the after distribution refinement would refine the extrude thing, which is not what you want. If you want refinement after distribution, you could use this code to do mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 -dm_distribute -dm_refine 2 and then call DMExtrude() by hand after that. Alternatively, I could make a pre-extrusion refinement option. Thanks, Matt > Thanks, > Bhargav > > /* refine the sequential mesh first */ > for (int r = 0; r < numRefinePre; ++r) { > DM rdm = NULL; > DMRefine(dm, MPI_COMM_WORLD, &rdm); > DMDestroy(&dm); > dm = rdm; > } > > /* project to a unit sphere */ > ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); > > /* create and distribute DM */ > ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); > if (dmDist) { > DMDestroy(&dm); > dm = dmDist; > } > > /* refine the mesh in parallel */ > for (int r = 0; r < numRefine; ++r) { > DM rdm = NULL; > DMRefine(dm, MPI_COMM_WORLD, &rdm); > DMDestroy(&dm); > dm = rdm; > } > > /* project to a unit sphere */ > ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); > > ierr = DMPlexExtrude(dm, numLayers-1, radialThickness, PETSC_FALSE, NULL, > PETSC_TRUE, &dm3D);CHKERRQ(ierr); > > if (dm3D) { > DMDestroy(&dm); > dm = dm3D; > } > > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: shell.png Type: image/png Size: 377130 bytes Desc: not available URL: From fdkong.jd at gmail.com Tue Nov 9 17:10:55 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Tue, 9 Nov 2021 16:10:55 -0700 Subject: [petsc-users] Error running make on MUMPS Message-ID: Hi All, We encountered a configuration error when running the PETSc configuration on a HPC system. Went through the log file, but could not find much. The log file was attached. Any thoughts? Thanks for your help, as always. Fande -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2516108 bytes Desc: not available URL: From bsmith at petsc.dev Tue Nov 9 18:41:10 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 9 Nov 2021 19:41:10 -0500 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: Message-ID: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> This version of MUMPS has a bug in its build system; it does not have all the dependencies on Fortran modules properly listed so Fortran files can get compiled too early causing "random" failures during some builds, especially on machines with lots of cores for compiling. I think you should be able to use --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 to get a patched version. Barry > On Nov 9, 2021, at 6:10 PM, Fande Kong wrote: > > Hi All, > > We encountered a configuration error when running the PETSc configuration on a HPC system. Went through the log file, but could not find much. The log file was attached. > > Any thoughts? > > Thanks for your help, as always. > > Fande > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yhcy1993 at gmail.com Wed Nov 10 02:42:06 2021 From: yhcy1993 at gmail.com (=?UTF-8?B?5LuT5a6H?=) Date: Wed, 10 Nov 2021 16:42:06 +0800 Subject: [petsc-users] Strange behavior of TS after setting hand-coded Jacobian In-Reply-To: References: <0C6ACBF3-F457-4BFD-AD19-8C455444748F@petsc.dev> <9CC15214-4601-4554-808F-C3E96DC3D34A@petsc.dev> <7564F0F0-BCB3-49C6-8D84-41B43D303D33@anl.gov> Message-ID: Thanks for your valuable advice. Your patience and recognition is highly appreciated. I will do some reading and think it over. Some tests will be performed in recent days. Regards. Yu Barry Smith ?2021?11?7??? ??11:15??? > > > I think you can use the following approach. Instead of making the boundary conditions, for example, > > V_1(t) = 0 > > you do the equivalent, > > v_1(t=0) = 0 and > > d V_1(t) > ---------- = 0 > dt > > > Now if you do backward Euler on this you get > > V_1^{n+1} - V_1^{n} > -------------------------- = 0 > Dt > > or in your residual form > > V_1^{n+1} - V_1^{n} > R_{V_1} = -------------------------- > Dt > > The Jacobian of this gives a 1/Dt on the diagonal for this row. (Similarly for the other problematic rows). > > Now your Jacobian is nonsingular and you can solve with it with no difficulties. > > Note that you get the same result if you think of the problem as a DAE because the residual equation is R_{V_1} = V_1^{n+1} and the derivative of this with respect to V_1^{n+1} is 1 (not zero). So you still end up with a nonzero on the diagonal, not zero. Regardless, you can keep these rows in the ODE form right hand side and form Jacobian, or you can eliminate them (analytically as Hong suggested) so the ODE solver does not see these rows but in either case the results are identical. > > Barry > > > On Nov 7, 2021, at 3:45 AM, ?? wrote: > > Thanks for your patient explanation, and sorry for this late reply. > > I do understand the manipulation skills that you mentioned, but this kind of manually substitution will not change the numerical difficulty of the problem. > > In fact, before I turn to PETSc, I have tried to form an ODE for this problem already, by utilizing the well-established functions like 'ode15s' and 'ode23' in MATLAB. But it failed to produce physically reasonable results. The key flaw lies in the treatment of the eigenvalue, namely \hat{\Lambda}_r denoted in the PDF file. Only implicit DAE formulation can fully embody its effect. This is why I finally decided to turn to use the DAE functionality in PETSc. > > I've excerpted some description from a monograph regarding this problem, hope it can clearly demonstrate the numerical requirement for solving it. > > Kee, R. J., Coltrin, M. E., Glarborg, P., & Zhu, H. (2017). Stagnation Flows. In Chemically Reacting Flow (2 ed., pp. 232). > > > > Zhang, Hong ?2021?11?1??? ??10:21??? >> >> You can avoid the zero-diagonal elements in your case. Just remove the boundary points and the \hat{\Lambda}_r from the solution vector Y (eq(17) in your PDE file). You can calculate them separately in your RHS function (when computing the residual vector for 2 <=i<=N-1). It is not necessary to form a DAE. You can actually form an ODE instead. >> >> Hong (Mr.) >> >> > On Oct 31, 2021, at 5:38 AM, ?? wrote: >> > >> > Thanks for your suggestions and experience sharing. >> > >> > But it seems that you have missed my point. This kind of zero-diagonal >> > element that I described earlier is somehow inevitable. >> > If interested in this problem, you may refer to the PDF file that was >> > attached in the previous post. >> > >> > Regards >> > >> > Yu >> > >> > Zhang, Hong ?2021?10?29??? ??10:05??? >> >> >> >> One way to avoid the zero element in Jacobian is to exclude the boundary point from the solution vector. I often do this for Dirichlet boundary conditions since the value at the boundary is given directly and does not need to be taken as a degree of freedom. >> >> >> >> Hong (Mr.) >> >> >> >> On Oct 28, 2021, at 9:49 PM, ?? wrote: >> >> >> >> Thanks for your careful inspection and thoughtful suggestions. >> >> >> >>> finite differencing may put a small non-zero value in that location due to numerical round-off >> >> >> >> I think your explanation is reasonable. This numerical round-off may somehow help to avoid this pivot issue. >> >> >> >> The structure of my jacobian matrix looks like this (generated by '-mat_view draw'): >> >> >> >> Analytically, the first diagonal element of the jacobian is indeed 0, as its corresponding residual function is solely determined from boundary condition of another variable. This seems a little bit wired but is mathematically well-posed. For more description about the background physics, please refer to attached PDF file for more detailed explanation on the discretization and boundary conditions. >> >> >> >> Actually, the jacobian matrix is not singular, but I do believe this numerical difficulty is caused by the zero-element in diagonal. >> >> In this regard, I've performed some trial and test. It seems that several methods have been worked out for this pivot issue: >> >> a) By setting '-pc_type svd', PETSC does not panic any more with my hand-coded jacobian, and converged solution is obtained. Efficiency is also preserved. >> >> b) By setting '-pc_type none', converged solution is also obtained, but it takes too many KSP iterations to converge per SNES step (usually hundreds), making the overall solution procedure very slow. >> >> >> >> Do you think these methods really solved this kind of pivot issue? Not by chance like the numerical round-off in finite difference previously. >> >> >> >> Regards >> >> >> >> Yu Cang >> >> >> >> Barry Smith ?2021?10?27??? ??9:43??? >> >>> >> >>> >> >>> You can run with -ksp_error_if_not_converged to get it to stop as soon as a linear solve fails to help track down the exact breaking point. >> >>> >> >>>> The problem under consideration contains an eigen-value to be solved, >> >>>> making the first diagonal element of the jacobian matrix being zero. >> >>>> From these outputs, it seems that the PC failed to factorize, which is >> >>>> caused by this 0 diagonal element. But I'm wondering why it works >> >>>> with jacobian matrix generated by finite-difference? >> >>> >> >>> Presumably your "exact" Jacobian puts a zero on the diagonal while the finite differencing may put a small non-zero value in that location due to numerical round-off. In that case even if the factorization succeeds it may produce an inaccurate solution if the value on the diagonal is very small. >> >>> >> >>> If your matrix is singular or cannot be factored with LU then you need to use a different solver for the linear system that will be robust to the zero on the diagonal. What is the structure of your Jacobian? (The analytic form). >> >>> >> >>> Barry >> >>> >> >>> >> >>>> On Oct 27, 2021, at 1:47 AM, ?? wrote: >> >>>> >> >>>> Thanks for your kind reply. >> >>>> >> >>>> Several comparison tests have been performed. Attached are execution >> >>>> output files. Below are corresponding descriptions. >> >>>> >> >>>> good.txt -- Run without hand-coded jacobian, solution converged, with >> >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >> >>>> -ksp_monitor_true_residual -ksp_converged_reason'; >> >>>> jac1.txt -- Run with hand-coded jacobian, does not converge, with >> >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >> >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian'; >> >>>> jac2.txt -- Run with hand-coded jacobian, does not converge, with >> >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >> >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >> >>>> -ksp_view'; >> >>>> jac3.txt -- Run with hand-coded jacobian, does not converge, with >> >>>> option '-ts_monitor -snes_monitor -snes_converged_reason >> >>>> -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian >> >>>> -ksp_view -ts_max_snes_failures -1 '; >> >>>> >> >>>> The problem under consideration contains an eigen-value to be solved, >> >>>> making the first diagonal element of the jacobian matrix being zero. >> >>>> From these outputs, it seems that the PC failed to factorize, which is >> >>>> caused by this 0 diagonal element. But I'm wondering why it works >> >>>> with jacobian matrix generated by finite-difference? Would employing >> >>>> DMDA for discretization be helpful? >> >>>> >> >>>> Regards >> >>>> >> >>>> Yu Cang >> >>>> >> >>>> Barry Smith ?2021?10?25??? ??10:50??? >> >>>>> >> >>>>> >> >>>>> It is definitely unexpected that -snes_test_jacobian verifies the Jacobian as matching but the solve process is completely different. >> >>>>> >> >>>>> Please run with -snes_monitor -snes_converged_reason -ksp_monitor_true_residual -ksp_converged_reason -snes_test_jacobian and send all the output >> >>>>> >> >>>>> Barry >> >>>>> >> >>>>> >> >>>>>> On Oct 25, 2021, at 9:53 AM, ?? wrote: >> >>>>>> >> >>>>>> I'm using TS to solve a set of DAE, which originates from a >> >>>>>> one-dimensional problem. The grid points are uniformly distributed. >> >>>>>> For simplicity, the DMDA is not employed for discretization. >> >>>>>> >> >>>>>> At first, only the residual function is prescribed through >> >>>>>> 'TSSetIFunction', and PETSC produces converged results. However, after >> >>>>>> providing hand-coded Jacobian through 'TSSetIJacobian', the internal >> >>>>>> SNES object fails (residual norm does not change), and TS reports >> >>>>>> 'DIVERGED_STEP_REJECTED'. >> >>>>>> >> >>>>>> I have tried to add the option '-snes_test_jacobian' to see if the >> >>>>>> hand-coded jacobian is somewhere wrong, but it shows '||J - >> >>>>>> Jfd||_F/||J||_F = 1.07488e-10, ||J - Jfd||_F = 2.14458e-07', >> >>>>>> indicating that the hand-coded jacobian is correct. >> >>>>>> >> >>>>>> Then, I added a monitor for the internal SNES object through >> >>>>>> 'SNESMonitorSet', in which the solution vector will be displayed at >> >>>>>> each iteration. It is interesting to find that, if the jacobian is not >> >>>>>> provided, meaning finite-difference is utilized for jacobian >> >>>>>> evaluation internally, the solution vector converges to steady >> >>>>>> solution and the SNES residual norm is reduced continuously. However, >> >>>>>> it turns out that, as long as the jacobian is provided, the solution >> >>>>>> vector will NEVER get changed! So the solution procedure stucked! >> >>>>>> >> >>>>>> This is quite strange! Hope to get some advice. >> >>>>>> PETSC version=3.14.6, program run in serial mode. >> >>>>>> >> >>>>>> Regards >> >>>>>> >> >>>>>> Yu Cang >> >>>>> >> >>>> >> >>> >> >> >> >> >> >> >> > From bhargav.subramanya at kaust.edu.sa Wed Nov 10 07:25:56 2021 From: bhargav.subramanya at kaust.edu.sa (Bhargav Subramanya) Date: Wed, 10 Nov 2021 16:25:56 +0300 Subject: [petsc-users] dmplex extrude in parallel In-Reply-To: References: Message-ID: Dear Matt, Thanks a lot for the reply. I am now able to generate the prismatic mesh properly. In the case of mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 -dm_distribute -dm_refine 2 and DMExtrude(), where I am ordering the extruded cells on the layers first; Is the mesh extruded in parallel for this case? The first time I distributed the mesh, it was only the spherical surface mesh. After dmplex extrude, the mesh changes. Do I need to redistribute the mesh again? Thanks, Bhargav On Wed, Nov 10, 2021 at 12:56 AM Matthew Knepley wrote: > On Tue, Nov 9, 2021 at 9:54 AM Bhargav Subramanya < > bhargav.subramanya at kaust.edu.sa> wrote: > >> Dear All, >> >> I want to generate a prismatic mesh from a base spherical surface mesh. I >> first refine the base icosahedron sequential mesh to some extent, >> distribute the mesh, continue the refinement in parallel, project the mesh >> to a unit sphere, and then finally extrude the mesh. The following is the >> code I am using. As the extrusion is performed in parallel, the extruded >> mesh seems to be broken as shown in the attached figure. May I know how to >> get an intact extruded mesh in parallel? Also, is it possible to make mesh >> refinement respect the spherical surface geometry, without having to >> project it using a function as shown below? >> > > I think you can do most of what you want from the command line. Here is a > simple code: > > #include > > int main(int argc, char **argv) > { > DM dm; > PetscErrorCode ierr; > > ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; > ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); > ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); > ierr = DMSetFromOptions(dm);CHKERRQ(ierr); > ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); > ierr = DMDestroy(&dm);CHKERRQ(ierr); > ierr = PetscFinalize(); > return ierr; > } > > that I run using > > mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 > -dm_extrude 5 -dm_plex_transform_extrude_thickness 0.1 -dm_distribute > -dm_view hdf5:mesh.h5 > > and get the attached picture. I am refining before distribution only here, > since the after distribution refinement would refine the extrude thing, > which is not what you want. > If you want refinement after distribution, you could use this code to do > > mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre > 3 -dm_distribute -dm_refine 2 > > and then call DMExtrude() by hand after that. Alternatively, I could make > a pre-extrusion refinement option. > > Thanks, > > Matt > > >> Thanks, >> Bhargav >> >> /* refine the sequential mesh first */ >> for (int r = 0; r < numRefinePre; ++r) { >> DM rdm = NULL; >> DMRefine(dm, MPI_COMM_WORLD, &rdm); >> DMDestroy(&dm); >> dm = rdm; >> } >> >> /* project to a unit sphere */ >> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >> >> /* create and distribute DM */ >> ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); >> if (dmDist) { >> DMDestroy(&dm); >> dm = dmDist; >> } >> >> /* refine the mesh in parallel */ >> for (int r = 0; r < numRefine; ++r) { >> DM rdm = NULL; >> DMRefine(dm, MPI_COMM_WORLD, &rdm); >> DMDestroy(&dm); >> dm = rdm; >> } >> >> /* project to a unit sphere */ >> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >> >> ierr = DMPlexExtrude(dm, numLayers-1, radialThickness, PETSC_FALSE, NULL, >> PETSC_TRUE, &dm3D);CHKERRQ(ierr); >> >> if (dm3D) { >> DMDestroy(&dm); >> dm = dm3D; >> } >> >> >> ------------------------------ >> This message and its contents, including attachments are intended solely >> for the original recipient. If you are not the intended recipient or have >> received this message in error, please notify me immediately and delete >> this message from your computer system. Any unauthorized use or >> distribution is prohibited. Please consider the environment before printing >> this email. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Nov 10 07:33:27 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Nov 2021 08:33:27 -0500 Subject: [petsc-users] dmplex extrude in parallel In-Reply-To: References: Message-ID: On Wed, Nov 10, 2021 at 8:26 AM Bhargav Subramanya < bhargav.subramanya at kaust.edu.sa> wrote: > Dear Matt, > > Thanks a lot for the reply. I am now able to generate the prismatic mesh > properly. > Cool. > In the case of mpiexec -n 2 ./meshtest -dm_plex_shape sphere > -dm_refine_pre 3 -dm_distribute -dm_refine 2 and DMExtrude(), where I am > ordering the extruded cells on the layers first; Is the mesh extruded in > parallel for this case? > Yes. > The first time I distributed the mesh, it was only the spherical surface > mesh. After dmplex extrude, the mesh changes. Do I need to redistribute the > mesh again? > Probably not. The balance factor will not change, meaning the relative sizes of the partitions. However, if you have a lot of layers, maybe you want to. It is easy, just call DMDistribute() again. Thanks, Matt > Thanks, > Bhargav > > On Wed, Nov 10, 2021 at 12:56 AM Matthew Knepley > wrote: > >> On Tue, Nov 9, 2021 at 9:54 AM Bhargav Subramanya < >> bhargav.subramanya at kaust.edu.sa> wrote: >> >>> Dear All, >>> >>> I want to generate a prismatic mesh from a base spherical surface mesh. >>> I first refine the base icosahedron sequential mesh to some extent, >>> distribute the mesh, continue the refinement in parallel, project the mesh >>> to a unit sphere, and then finally extrude the mesh. The following is the >>> code I am using. As the extrusion is performed in parallel, the extruded >>> mesh seems to be broken as shown in the attached figure. May I know how to >>> get an intact extruded mesh in parallel? Also, is it possible to make mesh >>> refinement respect the spherical surface geometry, without having to >>> project it using a function as shown below? >>> >> >> I think you can do most of what you want from the command line. Here is a >> simple code: >> >> #include >> >> int main(int argc, char **argv) >> { >> DM dm; >> PetscErrorCode ierr; >> >> ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; >> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); >> ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); >> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >> ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); >> ierr = DMDestroy(&dm);CHKERRQ(ierr); >> ierr = PetscFinalize(); >> return ierr; >> } >> >> that I run using >> >> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 >> -dm_extrude 5 -dm_plex_transform_extrude_thickness 0.1 -dm_distribute >> -dm_view hdf5:mesh.h5 >> >> and get the attached picture. I am refining before distribution only >> here, since the after distribution refinement would refine the extrude >> thing, which is not what you want. >> If you want refinement after distribution, you could use this code to do >> >> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre >> 3 -dm_distribute -dm_refine 2 >> >> and then call DMExtrude() by hand after that. Alternatively, I could make >> a pre-extrusion refinement option. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Bhargav >>> >>> /* refine the sequential mesh first */ >>> for (int r = 0; r < numRefinePre; ++r) { >>> DM rdm = NULL; >>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>> DMDestroy(&dm); >>> dm = rdm; >>> } >>> >>> /* project to a unit sphere */ >>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>> >>> /* create and distribute DM */ >>> ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); >>> if (dmDist) { >>> DMDestroy(&dm); >>> dm = dmDist; >>> } >>> >>> /* refine the mesh in parallel */ >>> for (int r = 0; r < numRefine; ++r) { >>> DM rdm = NULL; >>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>> DMDestroy(&dm); >>> dm = rdm; >>> } >>> >>> /* project to a unit sphere */ >>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>> >>> ierr = DMPlexExtrude(dm, numLayers-1, radialThickness, PETSC_FALSE, >>> NULL, PETSC_TRUE, &dm3D);CHKERRQ(ierr); >>> >>> if (dm3D) { >>> DMDestroy(&dm); >>> dm = dm3D; >>> } >>> >>> >>> ------------------------------ >>> This message and its contents, including attachments are intended solely >>> for the original recipient. If you are not the intended recipient or have >>> received this message in error, please notify me immediately and delete >>> this message from your computer system. Any unauthorized use or >>> distribution is prohibited. Please consider the environment before printing >>> this email. >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Vladislav.Pimanov at skoltech.ru Wed Nov 10 06:03:17 2021 From: Vladislav.Pimanov at skoltech.ru (Vladislav Pimanov) Date: Wed, 10 Nov 2021 12:03:17 +0000 Subject: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. Message-ID: <6553e88affc34cf28cc8cda4f71a22a4@skoltech.ru> Dear PETSc community, I wonder if you could give me a hint on how to compute the condition number of a preconditioned matrix in a proper way. I have a MatSchurComplement matrix S and a preconditioner P of the type PCSHELL (P is a diffusion matrix, which itself is inverted by KSPCG). I tried to compute the condition number of P^{-1}S "for free" during the outer PCG procedure using KSPComputeExtremeSingularValues() routine. Unfortunately, \sigma_min does not converge even if the solution is computed with very high precision. I also looked at SLEPc interface, but did not realised how PC should be included. Thanks! Sincerely, Vladislav Pimanov -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Nov 10 07:45:00 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Nov 2021 08:45:00 -0500 Subject: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. In-Reply-To: <6553e88affc34cf28cc8cda4f71a22a4@skoltech.ru> References: <6553e88affc34cf28cc8cda4f71a22a4@skoltech.ru> Message-ID: On Wed, Nov 10, 2021 at 8:42 AM Vladislav Pimanov < Vladislav.Pimanov at skoltech.ru> wrote: > Dear PETSc community, > > > I wonder if you could give me a hint on how to compute the condition > number of a preconditioned matrix in a proper way. > > I have a *MatSchurComplement* matrix S and a preconditioner P of the type > *PCSHELL* (P is a diffusion matrix, which itself is inverted by *KSPCG*). > > I tried to compute the condition number of P^{-1}S "for free" during > the outer PCG procedure using *KSPComputeExtremeSingularValues()* routine. > > Unfortunately, \sigma_min does not converge even if the solution is > computed with very high precision. > > I also looked at SLEPc interface, but did not realised how PC should be > included. > > You can do this at least two ways: 1) Make a MatShell for P^{-1} S. This is easy, but you will not be able to use any factorization-type PC on that matrix. 2) Solve instead the generalized EVP, S x = \lambda P x. Since you already have P^{-1}, this should work well. Thanks, Matt > Thanks! > > > Sincerely, > > Vladislav Pimanov > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From Vladislav.Pimanov at skoltech.ru Wed Nov 10 08:09:35 2021 From: Vladislav.Pimanov at skoltech.ru (Vladislav Pimanov) Date: Wed, 10 Nov 2021 14:09:35 +0000 Subject: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. In-Reply-To: References: <6553e88affc34cf28cc8cda4f71a22a4@skoltech.ru>, Message-ID: <0aea722c43a34006b03dd0eecde39d81@skoltech.ru> Great thanks, Matt! The second option is what I was looking for. Best Regards, Vlad ________________________________ ??: Matthew Knepley ??????????: 10 ?????? 2021 ?. 16:45:00 ????: Vladislav Pimanov ?????: petsc-users at mcs.anl.gov ????: Re: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. On Wed, Nov 10, 2021 at 8:42 AM Vladislav Pimanov > wrote: Dear PETSc community, I wonder if you could give me a hint on how to compute the condition number of a preconditioned matrix in a proper way. I have a MatSchurComplement matrix S and a preconditioner P of the type PCSHELL (P is a diffusion matrix, which itself is inverted by KSPCG). I tried to compute the condition number of P^{-1}S "for free" during the outer PCG procedure using KSPComputeExtremeSingularValues() routine. Unfortunately, \sigma_min does not converge even if the solution is computed with very high precision. I also looked at SLEPc interface, but did not realised how PC should be included. You can do this at least two ways: 1) Make a MatShell for P^{-1} S. This is easy, but you will not be able to use any factorization-type PC on that matrix. 2) Solve instead the generalized EVP, S x = \lambda P x. Since you already have P^{-1}, this should work well. Thanks, Matt Thanks! Sincerely, Vladislav Pimanov -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Wed Nov 10 08:45:55 2021 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 10 Nov 2021 09:45:55 -0500 Subject: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. In-Reply-To: <0aea722c43a34006b03dd0eecde39d81@skoltech.ru> References: <6553e88affc34cf28cc8cda4f71a22a4@skoltech.ru> <0aea722c43a34006b03dd0eecde39d81@skoltech.ru> Message-ID: <0610114B-D10E-4A1F-BA31-9BBF067B6F22@petsc.dev> > P is a diffusion matrix, which itself is inverted by KSPCG) This worries me. Unless solved to full precision the action of solving with CG is not a linear operator in the input variable b, this means that the action of your Schur complement is not a linear operator and so iterative eigenvalue algorithms may not work correctly (even if the linear system seems to be converging). I suggest you try a KSP tolerance of 10^-14 for the inner KSP solve when you attempt the eigenvalue computation. Barry > On Nov 10, 2021, at 9:09 AM, Vladislav Pimanov wrote: > > Great thanks, Matt! > The second option is what I was looking for. > > Best Regards, > Vlad > ??: Matthew Knepley > ??????????: 10 ?????? 2021 ?. 16:45:00 > ????: Vladislav Pimanov > ?????: petsc-users at mcs.anl.gov > ????: Re: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. > > On Wed, Nov 10, 2021 at 8:42 AM Vladislav Pimanov > wrote: > Dear PETSc community, > > > I wonder if you could give me a hint on how to compute the condition number of a preconditioned matrix in a proper way. > I have a MatSchurComplement matrix S and a preconditioner P of the type PCSHELL (P is a diffusion matrix, which itself is inverted by KSPCG). > I tried to compute the condition number of P^{-1}S "for free" during the outer PCG procedure using KSPComputeExtremeSingularValues() routine. > Unfortunately, \sigma_min does not converge even if the solution is computed with very high precision. > I also looked at SLEPc interface, but did not realised how PC should be included. > > You can do this at least two ways: > > 1) Make a MatShell for P^{-1} S. This is easy, but you will not be able to use any factorization-type PC on that matrix. > > 2) Solve instead the generalized EVP, S x = \lambda P x. Since you already have P^{-1}, this should work well. > > Thanks, > > Matt > Thanks! > > Sincerely, > Vladislav Pimanov > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Wed Nov 10 09:43:40 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Wed, 10 Nov 2021 08:43:40 -0700 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> Message-ID: Thanks, Barry, We will try this, and report back Fande On Tue, Nov 9, 2021 at 5:41 PM Barry Smith wrote: > > This version of MUMPS has a bug in its build system; it does not have > all the dependencies on Fortran modules properly listed so Fortran files > can get compiled too early causing "random" failures during some builds, > especially on machines with lots of cores for compiling. > > I think you should be able to use --download-mumps= > https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 > to get a patched version. > > Barry > > > On Nov 9, 2021, at 6:10 PM, Fande Kong wrote: > > Hi All, > > We encountered a configuration error when running the PETSc > configuration on a HPC system. Went through the log file, but could not > find much. The log file was attached. > > Any thoughts? > > Thanks for your help, as always. > > Fande > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anovak at anl.gov Wed Nov 10 13:00:44 2021 From: anovak at anl.gov (Novak, April) Date: Wed, 10 Nov 2021 19:00:44 +0000 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> Message-ID: Hi Barry, Thank you for your assistance - I?ve attached the latest configure.log. I still encounter issues building, though some of the MUMPS errors do seem to have been fixed with the --download-mumps-commit option. Do you have a recommendation for addressing these other errors? Thanks, -April From: Fande Kong Date: Wednesday, November 10, 2021 at 9:44 AM To: Barry Smith Cc: PETSc users list , Novak, April Subject: Re: [petsc-users] Error running make on MUMPS Thanks, Barry, We will try this, and report back Fande On Tue, Nov 9, 2021 at 5:41 PM Barry Smith > wrote: This version of MUMPS has a bug in its build system; it does not have all the dependencies on Fortran modules properly listed so Fortran files can get compiled too early causing "random" failures during some builds, especially on machines with lots of cores for compiling. I think you should be able to use --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 to get a patched version. Barry On Nov 9, 2021, at 6:10 PM, Fande Kong > wrote: Hi All, We encountered a configuration error when running the PETSc configuration on a HPC system. Went through the log file, but could not find much. The log file was attached. Any thoughts? Thanks for your help, as always. Fande -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2499919 bytes Desc: configure.log URL: From pierre at joliv.et Wed Nov 10 13:11:46 2021 From: pierre at joliv.et (Pierre Jolivet) Date: Wed, 10 Nov 2021 20:11:46 +0100 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> Message-ID: Hello April, You?ll need PETSc 3.16.0 to use --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 The MUMPS Makefile rule changed between 3.15.X (alllib) and 3.16.Y (all). Thanks, Pierre > On 10 Nov 2021, at 8:00 PM, Novak, April via petsc-users wrote: > > Hi Barry, > > Thank you for your assistance - I?ve attached the latest configure.log. I still encounter issues building, though some of the MUMPS errors do seem to have been fixed with the --download-mumps-commit option. Do you have a recommendation for addressing these other errors? > > Thanks, > > -April > > From: Fande Kong > Date: Wednesday, November 10, 2021 at 9:44 AM > To: Barry Smith > Cc: PETSc users list , Novak, April > Subject: Re: [petsc-users] Error running make on MUMPS > > Thanks, Barry, > > We will try this, and report back > > > Fande > > On Tue, Nov 9, 2021 at 5:41 PM Barry Smith > wrote: > > This version of MUMPS has a bug in its build system; it does not have all the dependencies on Fortran modules properly listed so Fortran files can get compiled too early causing "random" failures during some builds, especially on machines with lots of cores for compiling. > > I think you should be able to use --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 to get a patched version. > > Barry > > > > On Nov 9, 2021, at 6:10 PM, Fande Kong > wrote: > > Hi All, > > We encountered a configuration error when running the PETSc configuration on a HPC system. Went through the log file, but could not find much. The log file was attached. > > Any thoughts? > > Thanks for your help, as always. > > Fande > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Wed Nov 10 13:16:28 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Wed, 10 Nov 2021 13:16:28 -0600 (CST) Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> Message-ID: <61ccdb88-8636-cf84-69f5-eda0a1a6294b@mcs.anl.gov> You are using petsc-3.15.1 - and likely the mumps build change between then and current 3.16. Can you use latest PETSc release? If not - Suggest removing --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 options [and PETSC_ARCH] and going back to your old build. If it fails [as before] - retry with: --with-make-np=1 Satish On Wed, 10 Nov 2021, Novak, April via petsc-users wrote: > Hi Barry, > > Thank you for your assistance - I?ve attached the latest configure.log. I still encounter issues building, though some of the MUMPS errors do seem to have been fixed with the --download-mumps-commit option. Do you have a recommendation for addressing these other errors? > > Thanks, > > -April > > From: Fande Kong > Date: Wednesday, November 10, 2021 at 9:44 AM > To: Barry Smith > Cc: PETSc users list , Novak, April > Subject: Re: [petsc-users] Error running make on MUMPS > Thanks, Barry, > > We will try this, and report back > > > Fande > > On Tue, Nov 9, 2021 at 5:41 PM Barry Smith > wrote: > > This version of MUMPS has a bug in its build system; it does not have all the dependencies on Fortran modules properly listed so Fortran files can get compiled too early causing "random" failures during some builds, especially on machines with lots of cores for compiling. > > I think you should be able to use --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 to get a patched version. > > Barry > > > > On Nov 9, 2021, at 6:10 PM, Fande Kong > wrote: > > Hi All, > > We encountered a configuration error when running the PETSc configuration on a HPC system. Went through the log file, but could not find much. The log file was attached. > > Any thoughts? > > Thanks for your help, as always. > > Fande > > > > > > > > > From Eric.Chamberland at giref.ulaval.ca Wed Nov 10 14:08:22 2021 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Wed, 10 Nov 2021 15:08:22 -0500 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> <5751c2d4-e5d7-54d5-7f77-5bdb13c4a294@giref.ulaval.ca> Message-ID: Hi Matthew, we are continuing our developments around the overlap (funny to say... ) and we have a question about the "adjacency" used. It is said into the documentation of DMPlexDistributeOverlap that: "The user can control the definition of adjacency for the mesh using DMSetAdjacency(). They should choose the combination appropriate for the function representation on the mesh." We expected that this could influence whether the overlap computed includes cells that touch a domain by only one vertex for example. But we tried "all" combinations of useCone and useClosure on each DM we create in for the computation of the overlap (as in ex44.c we shared with you) but the computed overlap is always the same! However, we have not understood which "field" number to pass, so we used both PETSC_DEFAULT and DMSetBasicAdjacency but it makes no differences... Any clue what's wrong? Thanks, Eric On 2021-11-03 7:41 a.m., Matthew Knepley wrote: > On Tue, Nov 2, 2021 at 10:27 PM Eric Chamberland > > wrote: > > Hi Matthew, > > yes, it all works now! > > I modified the example to do a NaturalToGlobal then a > GlobalToNatural and test if we have the good result... > > The test would just need that we "force" the computed partition to > a given result, then we could check the result of the > NaturalToGlobal independently of the partitioner... > > Thanks a lot! > > No problem. Thanks for generating a great test. I will get this into > the CI tonight. > > ? Thanks, > > ? ? ?Matt > > Eric > > > On 2021-11-02 5:16 a.m., Matthew Knepley wrote: >> On Mon, Nov 1, 2021 at 5:24 PM Eric Chamberland >> > > wrote: >> >> Hi Matthew, >> >> ok, good news then! :) >> >> Meanwhile, I tested it the "other way" with >> DMPlexGlobalToNaturalBegin/End and it does not give a >> "sensed" result neither. >> >> Probably it does something like "GlobalToSequential" too? >> >> I am not sure how it worked before, but I think I have it fixed. >> I get the output you expect for the test. I force-pushed the branch >> so you will have to check it out again. >> >> ? Thanks! >> >> ? ? ?Matt >> >> Thanks! >> >> Eric >> >> >> On 2021-11-01 3:25 p.m., Matthew Knepley wrote: >>> On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland >>> >> > wrote: >>> >>> Hi Matthew, >>> >>> we do not know if DMPlexNaturalToGlobalBegin/End is >>> buggy or if it is our comprehension of what it should do >>> that is wrong... >>> >>> Would you just check if what we try to do from line 313 >>> to 356 is good or wrong? >>> >>> The expected result is that the global vector >>> "lGlobalVec" contains the element numbers from the >>> initial partition that have been put into "lNatVec". >>> >>> Thanks a lot for any insights!! >>> >>> Okay, I have found the problem.I confess to not actually >>> looking very closely at the implementation before. >>> GlobalToNatural was assuming that >>> the "natural" was also sequential, and looking at it all the >>> tests are eq-to-parallel distribution. I need to fix it for >>> a parallel natural state. Hopefully >>> it will not take me long. >>> >>> ? Thanks, >>> >>> ? ? ?Matt >>> >>> Eric >>> >>> >>> On 2021-10-27 2:32 p.m., Eric Chamberland wrote: >>>> >>>> Hi Matthew, >>>> >>>> we continued the example.? Now it must be our misuse of >>>> PETSc that produced the wrong result. >>>> >>>> As stated into the code: >>>> >>>> // The call to DMPlexNaturalToGlobalBegin/End does not >>>> produce our expected result... >>>> ? // In lGlobalVec, we expect to have: >>>> ? /* >>>> ?? * Process [0] >>>> ?? * 2. >>>> ?? * 4. >>>> ?? * 8. >>>> ?? * 3. >>>> ?? * 9. >>>> ?? * Process [1] >>>> ?? * 1. >>>> ?? * 5. >>>> ?? * 7. >>>> ?? * 0. >>>> ?? * 6. >>>> ?? * >>>> ?? * but we obtained: >>>> ?? * >>>> ?? * Process [0] >>>> ?? * 2. >>>> ?? * 4. >>>> ?? * 8. >>>> ?? * 0. >>>> ?? * 0. >>>> ?? * Process [1] >>>> ?? * 0. >>>> ?? * 0. >>>> ?? * 0. >>>> ?? * 0. >>>> ?? * 0. >>>> ?? */ >>>> >>>> (see attached ex44.c) >>>> >>>> Thanks, >>>> >>>> Eric >>>> >>>> On 2021-10-27 1:25 p.m., Eric Chamberland wrote: >>>>> >>>>> Great! >>>>> >>>>> Thanks Matthew, it is working for me up to that point! >>>>> >>>>> We are continuing the ex44.c and forward it to you at >>>>> the next blocking point... >>>>> >>>>> Eric >>>>> >>>>> On 2021-10-27 11:14 a.m., Matthew Knepley wrote: >>>>>> On Wed, Oct 27, 2021 at 8:29 AM Eric Chamberland >>>>>> >>>>> > wrote: >>>>>> >>>>>> Hi Matthew, >>>>>> >>>>>> the smallest mesh which crashes the code is a 2x5 >>>>>> mesh: >>>>>> >>>>>> See the modified ex44.c >>>>>> >>>>>> With smaller meshes(2x2, 2x4, etc), it passes...? >>>>>> But it bugs latter when I try to use >>>>>> DMPlexNaturalToGlobalBegin but let's keep that >>>>>> other problem for later... >>>>>> >>>>>> Thanks a lot for helping digging into this! :) >>>>>> >>>>>> I have made a small fix in this branch >>>>>> >>>>>> https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n >>>>>> >>>>>> >>>>>> It seems to run for me. Can you check it? >>>>>> >>>>>> ? Thanks, >>>>>> >>>>>> ? ? ?Matt >>>>>> >>>>>> Eric >>>>>> >>>>>> (sorry if you received this for a 2nd times, I >>>>>> have trouble with my mail) >>>>>> >>>>>> On 2021-10-26 4:35 p.m., Matthew Knepley wrote: >>>>>>> On Tue, Oct 26, 2021 at 1:35 PM Eric Chamberland >>>>>>> >>>>>> > wrote: >>>>>>> >>>>>>> Here is a screenshot of the partition I hard >>>>>>> coded (top) and vertices/element numbers (down): >>>>>>> >>>>>>> I have not yet modified the ex44.c example >>>>>>> to properly assign the coordinates... >>>>>>> >>>>>>> (but I would not have done it like it is in >>>>>>> the last version because the sCoords array >>>>>>> is the global array with global vertices number) >>>>>>> >>>>>>> I will have time to do this tomorrow... >>>>>>> >>>>>>> Maybe I can first try to reproduce all this >>>>>>> with a smaller mesh? >>>>>>> >>>>>>> >>>>>>> That might make it easier to find a problem. >>>>>>> >>>>>>> ? Thanks! >>>>>>> >>>>>>> ? ? ?Matt >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> On 2021-10-26 9:46 a.m., Matthew Knepley wrote: >>>>>>>> Okay, I ran it. Something seems off with >>>>>>>> the mesh. First, I cannot simply explain >>>>>>>> the partition. The number of shared >>>>>>>> vertices and edges >>>>>>>> does not seem to come from a straight cut. >>>>>>>> Second, the mesh look scrambled on output. >>>>>>>> >>>>>>>> ? Thanks, >>>>>>>> >>>>>>>> ? ? Matt >>>>>>>> >>>>>>>> On Sun, Oct 24, 2021 at 11:49 PM Eric >>>>>>>> Chamberland >>>>>>>> >>>>>>> > >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Matthew, >>>>>>>> >>>>>>>> ok, I started back from your ex44.c >>>>>>>> example and added the global array of >>>>>>>> coordinates. I just have to code the >>>>>>>> creation of the local coordinates now. >>>>>>>> >>>>>>>> Eric >>>>>>>> >>>>>>>> On 2021-10-20 6:55 p.m., Matthew >>>>>>>> Knepley wrote: >>>>>>>>> On Wed, Oct 20, 2021 at 3:06 PM Eric >>>>>>>>> Chamberland >>>>>>>>> >>>>>>>> > >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Matthew, >>>>>>>>> >>>>>>>>> we tried to reproduce the error in >>>>>>>>> a simple example. >>>>>>>>> >>>>>>>>> The context is the following: We >>>>>>>>> hard coded the mesh and initial >>>>>>>>> partition into the code (see >>>>>>>>> sConnectivity and >>>>>>>>> sInitialPartition) for 2 ranks and >>>>>>>>> try to create a section in order >>>>>>>>> to use the >>>>>>>>> DMPlexNaturalToGlobalBegin >>>>>>>>> function to retreive our initial >>>>>>>>> element numbers. >>>>>>>>> >>>>>>>>> Now the call to DMPlexDistribute >>>>>>>>> give different errors depending on >>>>>>>>> what type of component we ask the >>>>>>>>> field to be created.? For our >>>>>>>>> objective, we would like a global >>>>>>>>> field to be created on elements >>>>>>>>> only (like a P0 interpolation). >>>>>>>>> >>>>>>>>> We now have the following error >>>>>>>>> generated: >>>>>>>>> >>>>>>>>> [0]PETSC ERROR: >>>>>>>>> --------------------- Error >>>>>>>>> Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Petsc has >>>>>>>>> generated inconsistent data >>>>>>>>> [0]PETSC ERROR: Inconsistency in >>>>>>>>> indices, 18 should be 17 >>>>>>>>> [0]PETSC ERROR: See >>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>>> >>>>>>>>> for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Release >>>>>>>>> Version 3.15.0, Mar 30, 2021 >>>>>>>>> [0]PETSC ERROR: ./bug on a? named >>>>>>>>> rohan by ericc Wed Oct 20 14:52:36 >>>>>>>>> 2021 >>>>>>>>> [0]PETSC ERROR: Configure options >>>>>>>>> --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 >>>>>>>>> --with-mpi-compilers=1 >>>>>>>>> --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 >>>>>>>>> --with-cxx-dialect=C++14 >>>>>>>>> --with-make-np=12 >>>>>>>>> --with-shared-libraries=1 >>>>>>>>> --with-debugging=yes >>>>>>>>> --with-memalign=64 >>>>>>>>> --with-visibility=0 >>>>>>>>> --with-64-bit-indices=0 >>>>>>>>> --download-ml=yes >>>>>>>>> --download-mumps=yes >>>>>>>>> --download-superlu=yes >>>>>>>>> --download-hpddm=yes >>>>>>>>> --download-slepc=yes >>>>>>>>> --download-superlu_dist=yes >>>>>>>>> --download-parmetis=yes >>>>>>>>> --download-ptscotch=yes >>>>>>>>> --download-metis=yes >>>>>>>>> --download-strumpack=yes >>>>>>>>> --download-suitesparse=yes >>>>>>>>> --download-hypre=yes >>>>>>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>>>> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>>>> --with-scalapack=1 >>>>>>>>> --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include >>>>>>>>> --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>>>> -lmkl_scalapack_lp64 >>>>>>>>> -lmkl_blacs_openmpi_lp64" >>>>>>>>> [0]PETSC ERROR: #1 >>>>>>>>> PetscSFCreateSectionSF() at >>>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 >>>>>>>>> [0]PETSC ERROR: #2 >>>>>>>>> DMPlexCreateGlobalToNaturalSF() at >>>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 >>>>>>>>> [0]PETSC ERROR: #3 >>>>>>>>> DMPlexDistribute() at >>>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 >>>>>>>>> [0]PETSC ERROR: #4 main() at >>>>>>>>> bug_section.cc:159 >>>>>>>>> [0]PETSC ERROR: No PETSc Option >>>>>>>>> Table entries >>>>>>>>> [0]PETSC ERROR: >>>>>>>>> ----------------End of Error >>>>>>>>> Message -------send entire error >>>>>>>>> message to petsc-maint at mcs.anl.gov >>>>>>>>> ---------- >>>>>>>>> >>>>>>>>> Hope the attached code is >>>>>>>>> self-explaining, note that to make >>>>>>>>> it short, we have not included the >>>>>>>>> final part of it, just the buggy >>>>>>>>> part we are encountering right now... >>>>>>>>> >>>>>>>>> Thanks for your insights, >>>>>>>>> >>>>>>>>> Thanks for making the example. I >>>>>>>>> tweaked it slightly. I put in a test >>>>>>>>> case that just makes a parallel 7 x 10 >>>>>>>>> quad mesh. This works >>>>>>>>> fine. Thus I think it must be >>>>>>>>> something connected with the original >>>>>>>>> mesh. It is hard to get a handle on it >>>>>>>>> without the coordinates. >>>>>>>>> Do you think you could put the >>>>>>>>> coordinate array in? I have added the >>>>>>>>> code to load them (see attached file). >>>>>>>>> >>>>>>>>> ? Thanks, >>>>>>>>> >>>>>>>>> ?Matt >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> On 2021-10-06 9:23 p.m., Matthew >>>>>>>>> Knepley wrote: >>>>>>>>>> On Wed, Oct 6, 2021 at 5:43 PM >>>>>>>>>> Eric Chamberland >>>>>>>>>> >>>>>>>>> > >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Matthew, >>>>>>>>>> >>>>>>>>>> we tried to use that. Now, we >>>>>>>>>> discovered that: >>>>>>>>>> >>>>>>>>>> 1- even if we "ask" for >>>>>>>>>> sfNatural creation with >>>>>>>>>> DMSetUseNatural, it is not >>>>>>>>>> created because >>>>>>>>>> DMPlexCreateGlobalToNaturalSF >>>>>>>>>> looks for a "section": this >>>>>>>>>> is not documented in >>>>>>>>>> DMSetUseNaturalso we are >>>>>>>>>> asking ourselfs: "is this a >>>>>>>>>> permanent feature or a >>>>>>>>>> temporary situation?" >>>>>>>>>> >>>>>>>>>> I think explaining this will help >>>>>>>>>> clear up a lot. >>>>>>>>>> >>>>>>>>>> What the Natural2Global?map does >>>>>>>>>> is permute a solution vector into >>>>>>>>>> the ordering that it would have >>>>>>>>>> had prior to mesh distribution. >>>>>>>>>> Now, in order to do this >>>>>>>>>> permutation, I need to know the >>>>>>>>>> original (global) data layout. If >>>>>>>>>> it is not specified _before_ >>>>>>>>>> distribution, we >>>>>>>>>> cannot build the permutation. The >>>>>>>>>> section describes the data >>>>>>>>>> layout, so I need it before >>>>>>>>>> distribution. >>>>>>>>>> >>>>>>>>>> I cannot think of another way >>>>>>>>>> that you would implement this, >>>>>>>>>> but if you want something else, >>>>>>>>>> let me know. >>>>>>>>>> >>>>>>>>>> 2- We then tried to create a >>>>>>>>>> "section" in different >>>>>>>>>> manners: we took the code >>>>>>>>>> into the example >>>>>>>>>> petsc/src/dm/impls/plex/tests/ex15.c. >>>>>>>>>> However, we ended up with a >>>>>>>>>> segfault: >>>>>>>>>> >>>>>>>>>> corrupted size vs. prev_size >>>>>>>>>> [rohan:07297] *** Process >>>>>>>>>> received signal *** >>>>>>>>>> [rohan:07297] Signal: Aborted (6) >>>>>>>>>> [rohan:07297] Signal code: (-6) >>>>>>>>>> [rohan:07297] [ 0] >>>>>>>>>> /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >>>>>>>>>> [rohan:07297] [ 1] >>>>>>>>>> /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >>>>>>>>>> [rohan:07297] [ 2] >>>>>>>>>> /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >>>>>>>>>> [rohan:07297] [ 3] >>>>>>>>>> /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >>>>>>>>>> [rohan:07297] [ 4] >>>>>>>>>> /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >>>>>>>>>> [rohan:07297] [ 5] >>>>>>>>>> /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >>>>>>>>>> [rohan:07297] [ 6] >>>>>>>>>> /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >>>>>>>>>> [rohan:07297] [ 7] >>>>>>>>>> /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >>>>>>>>>> [rohan:07297] [ 8] >>>>>>>>>> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >>>>>>>>>> [rohan:07297] [ 9] >>>>>>>>>> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >>>>>>>>>> [rohan:07297] [10] >>>>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >>>>>>>>>> [rohan:07297] [11] >>>>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >>>>>>>>>> [rohan:07297] [12] >>>>>>>>>> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >>>>>>>>>> [rohan:07297] [13] >>>>>>>>>> /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >>>>>>>>>> >>>>>>>>>> [rohan:07297] [14] >>>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >>>>>>>>>> [rohan:07297] [15] >>>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >>>>>>>>>> [rohan:07297] [16] >>>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >>>>>>>>>> [rohan:07297] [17] >>>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >>>>>>>>>> [rohan:07297] [18] >>>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >>>>>>>>>> >>>>>>>>>> I am not sure what happened here, >>>>>>>>>> but if you could send a sample >>>>>>>>>> code, I will figure it out. >>>>>>>>>> >>>>>>>>>> If we do not create a >>>>>>>>>> section, the call to >>>>>>>>>> DMPlexDistribute is >>>>>>>>>> successful, but >>>>>>>>>> DMPlexGetGlobalToNaturalSF >>>>>>>>>> return a null SF pointer... >>>>>>>>>> >>>>>>>>>> Yes, it just ignores it in this >>>>>>>>>> case because it does not have a >>>>>>>>>> global layout. >>>>>>>>>> >>>>>>>>>> Here are the operations we >>>>>>>>>> are calling ( this is almost >>>>>>>>>> the code we are using, I just >>>>>>>>>> removed verifications and >>>>>>>>>> creation of the connectivity >>>>>>>>>> which use our parallel >>>>>>>>>> structure and code): >>>>>>>>>> >>>>>>>>>> =========== >>>>>>>>>> >>>>>>>>>> ? PetscInt* lCells????? = 0; >>>>>>>>>> ? PetscInt lNumCorners = 0; >>>>>>>>>> ? PetscInt lDimMail??? = 0; >>>>>>>>>> ? PetscInt lnumCells?? = 0; >>>>>>>>>> >>>>>>>>>> ? //At this point we create >>>>>>>>>> the cells for PETSc expected >>>>>>>>>> input for >>>>>>>>>> DMPlexBuildFromCellListParallel >>>>>>>>>> and set lNumCorners, lDimMail >>>>>>>>>> and lnumCells to correct values. >>>>>>>>>> ? ... >>>>>>>>>> >>>>>>>>>> ? DM lDMBete = 0 >>>>>>>>>> DMPlexCreate(lMPIComm,&lDMBete); >>>>>>>>>> >>>>>>>>>> DMSetDimension(lDMBete, >>>>>>>>>> lDimMail); >>>>>>>>>> >>>>>>>>>> DMPlexBuildFromCellListParallel(lDMBete, >>>>>>>>>> ????????????????????????????????? >>>>>>>>>> lnumCells, >>>>>>>>>> ????????????????????????????????? >>>>>>>>>> PETSC_DECIDE, >>>>>>>>>> pLectureElementsLocaux.reqNbTotalSommets(), >>>>>>>>>> ????????????????????????????????? >>>>>>>>>> lNumCorners, >>>>>>>>>> ????????????????????????????????? >>>>>>>>>> lCells, >>>>>>>>>> ????????????????????????????????? >>>>>>>>>> PETSC_NULL); >>>>>>>>>> >>>>>>>>>> ? DM lDMBeteInterp = 0; >>>>>>>>>> DMPlexInterpolate(lDMBete, >>>>>>>>>> &lDMBeteInterp); >>>>>>>>>> DMDestroy(&lDMBete); >>>>>>>>>> ? lDMBete = lDMBeteInterp; >>>>>>>>>> >>>>>>>>>> DMSetUseNatural(lDMBete,PETSC_TRUE); >>>>>>>>>> >>>>>>>>>> ? PetscSF lSFMigrationSansOvl >>>>>>>>>> = 0; >>>>>>>>>> ? PetscSF lSFMigrationOvl = 0; >>>>>>>>>> ? DM lDMDistribueSansOvl = 0; >>>>>>>>>> ? DM lDMAvecOverlap = 0; >>>>>>>>>> >>>>>>>>>> PetscPartitioner lPart; >>>>>>>>>> DMPlexGetPartitioner(lDMBete, >>>>>>>>>> &lPart); >>>>>>>>>> PetscPartitionerSetFromOptions(lPart); >>>>>>>>>> >>>>>>>>>> PetscSection section; >>>>>>>>>> PetscInt numFields?? = 1; >>>>>>>>>> PetscInt numBC?????? = 0; >>>>>>>>>> PetscInt numComp[1]? = {1}; >>>>>>>>>> PetscInt numDof[4]?? = {1, 0, >>>>>>>>>> 0, 0}; >>>>>>>>>> PetscInt bcFields[1] = {0}; >>>>>>>>>> IS bcPoints[1] = {NULL}; >>>>>>>>>> >>>>>>>>>> DMSetNumFields(lDMBete, >>>>>>>>>> numFields); >>>>>>>>>> >>>>>>>>>> DMPlexCreateSection(lDMBete, >>>>>>>>>> NULL, numComp, numDof, numBC, >>>>>>>>>> bcFields, bcPoints, NULL, >>>>>>>>>> NULL, §ion); >>>>>>>>>> DMSetLocalSection(lDMBete, >>>>>>>>>> section); >>>>>>>>>> >>>>>>>>>> DMPlexDistribute(lDMBete, 0, >>>>>>>>>> &lSFMigrationSansOvl, >>>>>>>>>> &lDMDistribueSansOvl); // >>>>>>>>>> segfault! >>>>>>>>>> >>>>>>>>>> =========== >>>>>>>>>> >>>>>>>>>> So we have other >>>>>>>>>> question/remarks: >>>>>>>>>> >>>>>>>>>> 3- Maybe PETSc expect >>>>>>>>>> something specific that is >>>>>>>>>> missing/not verified: for >>>>>>>>>> example, we didn't gave any >>>>>>>>>> coordinates since we just >>>>>>>>>> want to partition and compute >>>>>>>>>> overlap for the mesh... and >>>>>>>>>> then recover our element >>>>>>>>>> numbers in a "simple way" >>>>>>>>>> >>>>>>>>>> 4- We are telling ourselves >>>>>>>>>> it is somewhat a "big price >>>>>>>>>> to pay" to have to build an >>>>>>>>>> unused section to have the >>>>>>>>>> global to natural ordering >>>>>>>>>> set ?? Could this requirement >>>>>>>>>> be avoided? >>>>>>>>>> >>>>>>>>>> I don't think so. There would >>>>>>>>>> have to be _some_ way of >>>>>>>>>> describing your data layout in >>>>>>>>>> terms of mesh points, and I do >>>>>>>>>> not see how you could use less >>>>>>>>>> memory doing that. >>>>>>>>>> >>>>>>>>>> 5- Are there any improvement >>>>>>>>>> towards our usages in 3.16 >>>>>>>>>> release? >>>>>>>>>> >>>>>>>>>> Let me try and run the code above. >>>>>>>>>> >>>>>>>>>> ? Thanks, >>>>>>>>>> >>>>>>>>>> ?Matt >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2021-09-29 7:39 p.m., >>>>>>>>>> Matthew Knepley wrote: >>>>>>>>>>> On Wed, Sep 29, 2021 at 5:18 >>>>>>>>>>> PM Eric Chamberland >>>>>>>>>>> >>>>>>>>>> > >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I come back with >>>>>>>>>>> _almost_ the original >>>>>>>>>>> question: >>>>>>>>>>> >>>>>>>>>>> I would like to add an >>>>>>>>>>> integer information >>>>>>>>>>> (*our* original element >>>>>>>>>>> number, not petsc one) >>>>>>>>>>> on each element of the >>>>>>>>>>> DMPlex I create with >>>>>>>>>>> DMPlexBuildFromCellListParallel. >>>>>>>>>>> >>>>>>>>>>> I would like this >>>>>>>>>>> interger to be >>>>>>>>>>> distribruted by or the >>>>>>>>>>> same way >>>>>>>>>>> DMPlexDistribute >>>>>>>>>>> distribute the mesh. >>>>>>>>>>> >>>>>>>>>>> Is it possible to do this? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I think we already have >>>>>>>>>>> support for what you want. >>>>>>>>>>> If you call >>>>>>>>>>> >>>>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> before DMPlexDistribute(), >>>>>>>>>>> it will compute a PetscSF >>>>>>>>>>> encoding the global to >>>>>>>>>>> natural map. You >>>>>>>>>>> can get it with >>>>>>>>>>> >>>>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> and use it with >>>>>>>>>>> >>>>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Is this sufficient? >>>>>>>>>>> >>>>>>>>>>> ? Thanks, >>>>>>>>>>> >>>>>>>>>>> ? ? ?Matt >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Eric >>>>>>>>>>> >>>>>>>>>>> On 2021-07-14 1:18 p.m., >>>>>>>>>>> Eric Chamberland wrote: >>>>>>>>>>> > Hi, >>>>>>>>>>> > >>>>>>>>>>> > I want to use >>>>>>>>>>> DMPlexDistribute from >>>>>>>>>>> PETSc for computing >>>>>>>>>>> overlapping >>>>>>>>>>> > and play with the >>>>>>>>>>> different partitioners >>>>>>>>>>> supported. >>>>>>>>>>> > >>>>>>>>>>> > However, after calling >>>>>>>>>>> DMPlexDistribute, I >>>>>>>>>>> noticed the elements are >>>>>>>>>>> > renumbered and then >>>>>>>>>>> the original number is lost. >>>>>>>>>>> > >>>>>>>>>>> > What would be the best >>>>>>>>>>> way to keep track of the >>>>>>>>>>> element renumbering? >>>>>>>>>>> > >>>>>>>>>>> > a) Adding an optional >>>>>>>>>>> parameter to let the >>>>>>>>>>> user retrieve a vector or >>>>>>>>>>> > "IS" giving the old >>>>>>>>>>> number? >>>>>>>>>>> > >>>>>>>>>>> > b) Adding a DMLabel >>>>>>>>>>> (seems a wrong good >>>>>>>>>>> solution) >>>>>>>>>>> > >>>>>>>>>>> > c) Other idea? >>>>>>>>>>> > >>>>>>>>>>> > Of course, I don't >>>>>>>>>>> want to loose >>>>>>>>>>> performances with the >>>>>>>>>>> need of this >>>>>>>>>>> > "mapping"... >>>>>>>>>>> > >>>>>>>>>>> > Thanks, >>>>>>>>>>> > >>>>>>>>>>> > Eric >>>>>>>>>>> > >>>>>>>>>>> -- >>>>>>>>>>> Eric Chamberland, ing., >>>>>>>>>>> M. Ing >>>>>>>>>>> Professionnel de recherche >>>>>>>>>>> GIREF/Universit? Laval >>>>>>>>>>> (418) 656-2131 poste 41 >>>>>>>>>>> 22 42 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take >>>>>>>>>>> for granted before they >>>>>>>>>>> begin their experiments is >>>>>>>>>>> infinitely more interesting >>>>>>>>>>> than any results to which >>>>>>>>>>> their experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>>> Professionnel de recherche >>>>>>>>>> GIREF/Universit? Laval >>>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for >>>>>>>>>> granted before they begin their >>>>>>>>>> experiments is infinitely more >>>>>>>>>> interesting than any results to >>>>>>>>>> which their experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Universit? Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for >>>>>>>>> granted before they begin their >>>>>>>>> experiments is infinitely more >>>>>>>>> interesting than any results to which >>>>>>>>> their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Universit? Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted >>>>>>>> before they begin their experiments is >>>>>>>> infinitely more interesting than any >>>>>>>> results to which their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Eric Chamberland, ing., M. Ing >>>>>>> Professionnel de recherche >>>>>>> GIREF/Universit? Laval >>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before >>>>>>> they begin their experiments is infinitely more >>>>>>> interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Universit? Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they >>>>>> begin their experiments is infinitely more >>>>>> interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Universit? Laval >>>>> (418) 656-2131 poste 41 22 42 >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin >>> their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to >> which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which > their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Universit? Laval (418) 656-2131 poste 41 22 42 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbnbhlbilhmjdpfg.png Type: image/png Size: 42972 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eejjfmbjimlkboec.png Type: image/png Size: 87901 bytes Desc: not available URL: From knepley at gmail.com Wed Nov 10 14:26:27 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 10 Nov 2021 15:26:27 -0500 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> <5751c2d4-e5d7-54d5-7f77-5bdb13c4a294@giref.ulaval.ca> Message-ID: On Wed, Nov 10, 2021 at 3:08 PM Eric Chamberland < Eric.Chamberland at giref.ulaval.ca> wrote: > Hi Matthew, > > we are continuing our developments around the overlap (funny to say... ) > and we have a question about the "adjacency" used. > > It is said into the documentation of DMPlexDistributeOverlap that: > > "The user can control the definition of adjacency for the mesh using > DMSetAdjacency(). They should choose the combination appropriate for the > function representation on the mesh." > > We expected that this could influence whether the overlap computed > includes cells that touch a domain by only one vertex for example. > > But we tried "all" combinations of useCone and useClosure on each DM we > create in for the computation of the overlap (as in ex44.c we shared with > you) but the computed overlap is always the same! > > However, we have not understood which "field" number to pass, so we used > both PETSC_DEFAULT and DMSetBasicAdjacency but it makes no differences... > > Any clue what's wrong? > Okay, so the PETSC_DEFAULT one is what is used for topology. There are two separate uses for adjacency info. First, for this kind of topological extension, and the second is for calculating Jacobians. You should be able to see the difference between FEM and FVM in just the case of a 2x2 mesh where each process gets 1 cell. With FEM, the overlap is the whole mesh, whereas with FVM it leaves out the diagonal partner. Thanks, Matt > Thanks, > > Eric > > > On 2021-11-03 7:41 a.m., Matthew Knepley wrote: > > On Tue, Nov 2, 2021 at 10:27 PM Eric Chamberland < > Eric.Chamberland at giref.ulaval.ca> wrote: > >> Hi Matthew, >> >> yes, it all works now! >> >> I modified the example to do a NaturalToGlobal then a GlobalToNatural and >> test if we have the good result... >> >> The test would just need that we "force" the computed partition to a >> given result, then we could check the result of the NaturalToGlobal >> independently of the partitioner... >> >> Thanks a lot! >> > No problem. Thanks for generating a great test. I will get this into the > CI tonight. > > Thanks, > > Matt > >> Eric >> >> >> On 2021-11-02 5:16 a.m., Matthew Knepley wrote: >> >> On Mon, Nov 1, 2021 at 5:24 PM Eric Chamberland < >> Eric.Chamberland at giref.ulaval.ca> wrote: >> >>> Hi Matthew, >>> >>> ok, good news then! :) >>> >>> Meanwhile, I tested it the "other way" with >>> DMPlexGlobalToNaturalBegin/End and it does not give a "sensed" result >>> neither. >>> >>> Probably it does something like "GlobalToSequential" too? >>> >> I am not sure how it worked before, but I think I have it fixed. I get >> the output you expect for the test. I force-pushed the branch >> so you will have to check it out again. >> >> Thanks! >> >> Matt >> >>> Thanks! >>> >>> Eric >>> >>> >>> On 2021-11-01 3:25 p.m., Matthew Knepley wrote: >>> >>> On Sun, Oct 31, 2021 at 10:07 AM Eric Chamberland < >>> Eric.Chamberland at giref.ulaval.ca> wrote: >>> >>>> Hi Matthew, >>>> >>>> we do not know if DMPlexNaturalToGlobalBegin/End is buggy or if it is >>>> our comprehension of what it should do that is wrong... >>>> >>>> Would you just check if what we try to do from line 313 to 356 is good >>>> or wrong? >>>> >>>> The expected result is that the global vector "lGlobalVec" contains the >>>> element numbers from the initial partition that have been put into >>>> "lNatVec". >>>> >>>> Thanks a lot for any insights!! >>>> >>> Okay, I have found the problem.I confess to not actually looking very >>> closely at the implementation before. GlobalToNatural was assuming that >>> the "natural" was also sequential, and looking at it all the tests are >>> eq-to-parallel distribution. I need to fix it for a parallel natural state. >>> Hopefully >>> it will not take me long. >>> >>> Thanks, >>> >>> Matt >>> >>>> Eric >>>> >>>> >>>> On 2021-10-27 2:32 p.m., Eric Chamberland wrote: >>>> >>>> Hi Matthew, >>>> >>>> we continued the example. Now it must be our misuse of PETSc that >>>> produced the wrong result. >>>> >>>> As stated into the code: >>>> >>>> // The call to DMPlexNaturalToGlobalBegin/End does not produce our >>>> expected result... >>>> // In lGlobalVec, we expect to have: >>>> /* >>>> * Process [0] >>>> * 2. >>>> * 4. >>>> * 8. >>>> * 3. >>>> * 9. >>>> * Process [1] >>>> * 1. >>>> * 5. >>>> * 7. >>>> * 0. >>>> * 6. >>>> * >>>> * but we obtained: >>>> * >>>> * Process [0] >>>> * 2. >>>> * 4. >>>> * 8. >>>> * 0. >>>> * 0. >>>> * Process [1] >>>> * 0. >>>> * 0. >>>> * 0. >>>> * 0. >>>> * 0. >>>> */ >>>> >>>> (see attached ex44.c) >>>> >>>> Thanks, >>>> >>>> Eric >>>> On 2021-10-27 1:25 p.m., Eric Chamberland wrote: >>>> >>>> Great! >>>> >>>> Thanks Matthew, it is working for me up to that point! >>>> >>>> We are continuing the ex44.c and forward it to you at the next blocking >>>> point... >>>> >>>> Eric >>>> On 2021-10-27 11:14 a.m., Matthew Knepley wrote: >>>> >>>> On Wed, Oct 27, 2021 at 8:29 AM Eric Chamberland < >>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>> >>>>> Hi Matthew, >>>>> >>>>> the smallest mesh which crashes the code is a 2x5 mesh: >>>>> >>>>> See the modified ex44.c >>>>> >>>>> With smaller meshes(2x2, 2x4, etc), it passes... But it bugs latter >>>>> when I try to use DMPlexNaturalToGlobalBegin but let's keep that other >>>>> problem for later... >>>>> >>>>> Thanks a lot for helping digging into this! :) >>>>> >>>> I have made a small fix in this branch >>>> >>>> https://gitlab.com/petsc/petsc/-/commits/knepley/fix-plex-g2n >>>> >>>> It seems to run for me. Can you check it? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Eric >>>>> >>>>> (sorry if you received this for a 2nd times, I have trouble with my >>>>> mail) >>>>> On 2021-10-26 4:35 p.m., Matthew Knepley wrote: >>>>> >>>>> On Tue, Oct 26, 2021 at 1:35 PM Eric Chamberland < >>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>> >>>>>> Here is a screenshot of the partition I hard coded (top) and >>>>>> vertices/element numbers (down): >>>>>> >>>>>> I have not yet modified the ex44.c example to properly assign the >>>>>> coordinates... >>>>>> >>>>>> (but I would not have done it like it is in the last version because >>>>>> the sCoords array is the global array with global vertices number) >>>>>> >>>>>> I will have time to do this tomorrow... >>>>>> >>>>>> Maybe I can first try to reproduce all this with a smaller mesh? >>>>>> >>>>> >>>>> That might make it easier to find a problem. >>>>> >>>>> Thanks! >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Eric >>>>>> On 2021-10-26 9:46 a.m., Matthew Knepley wrote: >>>>>> >>>>>> Okay, I ran it. Something seems off with the mesh. First, I cannot >>>>>> simply explain the partition. The number of shared vertices and edges >>>>>> does not seem to come from a straight cut. Second, the mesh look >>>>>> scrambled on output. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> On Sun, Oct 24, 2021 at 11:49 PM Eric Chamberland < >>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>> >>>>>>> Hi Matthew, >>>>>>> >>>>>>> ok, I started back from your ex44.c example and added the global >>>>>>> array of coordinates. I just have to code the creation of the local >>>>>>> coordinates now. >>>>>>> >>>>>>> Eric >>>>>>> On 2021-10-20 6:55 p.m., Matthew Knepley wrote: >>>>>>> >>>>>>> On Wed, Oct 20, 2021 at 3:06 PM Eric Chamberland < >>>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>>> >>>>>>>> Hi Matthew, >>>>>>>> >>>>>>>> we tried to reproduce the error in a simple example. >>>>>>>> >>>>>>>> The context is the following: We hard coded the mesh and initial >>>>>>>> partition into the code (see sConnectivity and sInitialPartition) for 2 >>>>>>>> ranks and try to create a section in order to use the >>>>>>>> DMPlexNaturalToGlobalBegin function to retreive our initial element numbers. >>>>>>>> >>>>>>>> Now the call to DMPlexDistribute give different errors depending on >>>>>>>> what type of component we ask the field to be created. For our objective, >>>>>>>> we would like a global field to be created on elements only (like a P0 >>>>>>>> interpolation). >>>>>>>> >>>>>>>> We now have the following error generated: >>>>>>>> >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Petsc has generated inconsistent data >>>>>>>> [0]PETSC ERROR: Inconsistency in indices, 18 should be 17 >>>>>>>> [0]PETSC ERROR: See >>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble >>>>>>>> shooting. >>>>>>>> [0]PETSC ERROR: Petsc Release Version 3.15.0, Mar 30, 2021 >>>>>>>> [0]PETSC ERROR: ./bug on a named rohan by ericc Wed Oct 20 >>>>>>>> 14:52:36 2021 >>>>>>>> [0]PETSC ERROR: Configure options >>>>>>>> --prefix=/opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7 --with-mpi-compilers=1 >>>>>>>> --with-mpi-dir=/opt/openmpi-4.1.0_gcc7 --with-cxx-dialect=C++14 >>>>>>>> --with-make-np=12 --with-shared-libraries=1 --with-debugging=yes >>>>>>>> --with-memalign=64 --with-visibility=0 --with-64-bit-indices=0 >>>>>>>> --download-ml=yes --download-mumps=yes --download-superlu=yes >>>>>>>> --download-hpddm=yes --download-slepc=yes --download-superlu_dist=yes >>>>>>>> --download-parmetis=yes --download-ptscotch=yes --download-metis=yes >>>>>>>> --download-strumpack=yes --download-suitesparse=yes --download-hypre=yes >>>>>>>> --with-blaslapack-dir=/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>>> --with-mkl_pardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>>> --with-mkl_cpardiso-dir=/opt/intel/oneapi/mkl/2021.1.1/env/.. >>>>>>>> --with-scalapack=1 >>>>>>>> --with-scalapack-include=/opt/intel/oneapi/mkl/2021.1.1/env/../include >>>>>>>> --with-scalapack-lib="-L/opt/intel/oneapi/mkl/2021.1.1/env/../lib/intel64 >>>>>>>> -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64" >>>>>>>> [0]PETSC ERROR: #1 PetscSFCreateSectionSF() at >>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/vec/is/sf/utils/sfutils.c:409 >>>>>>>> [0]PETSC ERROR: #2 DMPlexCreateGlobalToNaturalSF() at >>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexnatural.c:184 >>>>>>>> [0]PETSC ERROR: #3 DMPlexDistribute() at >>>>>>>> /tmp/ompi-opt/petsc-3.15.0-debug/src/dm/impls/plex/plexdistribute.c:1733 >>>>>>>> [0]PETSC ERROR: #4 main() at bug_section.cc:159 >>>>>>>> [0]PETSC ERROR: No PETSc Option Table entries >>>>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>>>>> entire error message to petsc-maint at mcs.anl.gov---------- >>>>>>>> >>>>>>>> Hope the attached code is self-explaining, note that to make it >>>>>>>> short, we have not included the final part of it, just the buggy part we >>>>>>>> are encountering right now... >>>>>>>> >>>>>>>> Thanks for your insights, >>>>>>>> >>>>>>> Thanks for making the example. I tweaked it slightly. I put in a >>>>>>> test case that just makes a parallel 7 x 10 quad mesh. This works >>>>>>> fine. Thus I think it must be something connected with the original >>>>>>> mesh. It is hard to get a handle on it without the coordinates. >>>>>>> Do you think you could put the coordinate array in? I have added the >>>>>>> code to load them (see attached file). >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>>> Eric >>>>>>>> On 2021-10-06 9:23 p.m., Matthew Knepley wrote: >>>>>>>> >>>>>>>> On Wed, Oct 6, 2021 at 5:43 PM Eric Chamberland < >>>>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>>>> >>>>>>>>> Hi Matthew, >>>>>>>>> >>>>>>>>> we tried to use that. Now, we discovered that: >>>>>>>>> >>>>>>>>> 1- even if we "ask" for sfNatural creation with DMSetUseNatural, >>>>>>>>> it is not created because DMPlexCreateGlobalToNaturalSF looks for a >>>>>>>>> "section": this is not documented in DMSetUseNaturalso we are asking >>>>>>>>> ourselfs: "is this a permanent feature or a temporary situation?" >>>>>>>>> >>>>>>>> I think explaining this will help clear up a lot. >>>>>>>> >>>>>>>> What the Natural2Global map does is permute a solution vector into >>>>>>>> the ordering that it would have had prior to mesh distribution. >>>>>>>> Now, in order to do this permutation, I need to know the original >>>>>>>> (global) data layout. If it is not specified _before_ distribution, we >>>>>>>> cannot build the permutation. The section describes the data >>>>>>>> layout, so I need it before distribution. >>>>>>>> >>>>>>>> I cannot think of another way that you would implement this, but if >>>>>>>> you want something else, let me know. >>>>>>>> >>>>>>>>> 2- We then tried to create a "section" in different manners: we >>>>>>>>> took the code into the example petsc/src/dm/impls/plex/tests/ex15.c. >>>>>>>>> However, we ended up with a segfault: >>>>>>>>> >>>>>>>>> corrupted size vs. prev_size >>>>>>>>> [rohan:07297] *** Process received signal *** >>>>>>>>> [rohan:07297] Signal: Aborted (6) >>>>>>>>> [rohan:07297] Signal code: (-6) >>>>>>>>> [rohan:07297] [ 0] /lib64/libpthread.so.0(+0x13f80)[0x7f6f13be3f80] >>>>>>>>> [rohan:07297] [ 1] /lib64/libc.so.6(gsignal+0x10b)[0x7f6f109b718b] >>>>>>>>> [rohan:07297] [ 2] /lib64/libc.so.6(abort+0x175)[0x7f6f109b8585] >>>>>>>>> [rohan:07297] [ 3] /lib64/libc.so.6(+0x7e2f7)[0x7f6f109fb2f7] >>>>>>>>> [rohan:07297] [ 4] /lib64/libc.so.6(+0x857ea)[0x7f6f10a027ea] >>>>>>>>> [rohan:07297] [ 5] /lib64/libc.so.6(+0x86036)[0x7f6f10a03036] >>>>>>>>> [rohan:07297] [ 6] /lib64/libc.so.6(+0x861a3)[0x7f6f10a031a3] >>>>>>>>> [rohan:07297] [ 7] /lib64/libc.so.6(+0x88740)[0x7f6f10a05740] >>>>>>>>> [rohan:07297] [ 8] >>>>>>>>> /lib64/libc.so.6(__libc_malloc+0x1b8)[0x7f6f10a070c8] >>>>>>>>> [rohan:07297] [ 9] >>>>>>>>> /lib64/libc.so.6(__backtrace_symbols+0x134)[0x7f6f10a8b064] >>>>>>>>> [rohan:07297] [10] >>>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z12reqBacktraceRNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4e)[0x4538ce] >>>>>>>>> [rohan:07297] [11] >>>>>>>>> /home/mefpp_ericc/GIREF/bin/MEF++.dev(_Z15attacheDebuggerv+0x120)[0x4523c0] >>>>>>>>> [rohan:07297] [12] >>>>>>>>> /home/mefpp_ericc/GIREF/lib/libgiref_dev_Util.so(traitementSignal+0x612)[0x7f6f28f503a2] >>>>>>>>> [rohan:07297] [13] /lib64/libc.so.6(+0x3a210)[0x7f6f109b7210] >>>>>>>>> >>>>>>>>> [rohan:07297] [14] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscTrMallocDefault+0x6fd)[0x7f6f22f1b8ed] >>>>>>>>> [rohan:07297] [15] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscMallocA+0x5cd)[0x7f6f22f19c2d] >>>>>>>>> [rohan:07297] [16] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(PetscSFCreateSectionSF+0xb48)[0x7f6f23268e18] >>>>>>>>> [rohan:07297] [17] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexCreateGlobalToNaturalSF+0x13b2)[0x7f6f241a5602] >>>>>>>>> [rohan:07297] [18] >>>>>>>>> /opt/petsc-3.15.0_debug_openmpi-4.1.0_gcc7/lib/libpetsc.so.3.15(DMPlexDistribute+0x39b1)[0x7f6f23fdca21] >>>>>>>>> >>>>>>>> I am not sure what happened here, but if you could send a sample >>>>>>>> code, I will figure it out. >>>>>>>> >>>>>>>>> If we do not create a section, the call to DMPlexDistribute is >>>>>>>>> successful, but DMPlexGetGlobalToNaturalSF return a null SF pointer... >>>>>>>>> >>>>>>>> Yes, it just ignores it in this case because it does not have a >>>>>>>> global layout. >>>>>>>> >>>>>>>>> Here are the operations we are calling ( this is almost the code >>>>>>>>> we are using, I just removed verifications and creation of the connectivity >>>>>>>>> which use our parallel structure and code): >>>>>>>>> >>>>>>>>> =========== >>>>>>>>> >>>>>>>>> PetscInt* lCells = 0; >>>>>>>>> PetscInt lNumCorners = 0; >>>>>>>>> PetscInt lDimMail = 0; >>>>>>>>> PetscInt lnumCells = 0; >>>>>>>>> >>>>>>>>> //At this point we create the cells for PETSc expected input for >>>>>>>>> DMPlexBuildFromCellListParallel and set lNumCorners, lDimMail and lnumCells >>>>>>>>> to correct values. >>>>>>>>> ... >>>>>>>>> >>>>>>>>> DM lDMBete = 0 >>>>>>>>> DMPlexCreate(lMPIComm,&lDMBete); >>>>>>>>> >>>>>>>>> DMSetDimension(lDMBete, lDimMail); >>>>>>>>> >>>>>>>>> DMPlexBuildFromCellListParallel(lDMBete, >>>>>>>>> lnumCells, >>>>>>>>> PETSC_DECIDE, >>>>>>>>> >>>>>>>>> pLectureElementsLocaux.reqNbTotalSommets(), >>>>>>>>> lNumCorners, >>>>>>>>> lCells, >>>>>>>>> PETSC_NULL); >>>>>>>>> >>>>>>>>> DM lDMBeteInterp = 0; >>>>>>>>> DMPlexInterpolate(lDMBete, &lDMBeteInterp); >>>>>>>>> DMDestroy(&lDMBete); >>>>>>>>> lDMBete = lDMBeteInterp; >>>>>>>>> >>>>>>>>> DMSetUseNatural(lDMBete,PETSC_TRUE); >>>>>>>>> >>>>>>>>> PetscSF lSFMigrationSansOvl = 0; >>>>>>>>> PetscSF lSFMigrationOvl = 0; >>>>>>>>> DM lDMDistribueSansOvl = 0; >>>>>>>>> DM lDMAvecOverlap = 0; >>>>>>>>> >>>>>>>>> PetscPartitioner lPart; >>>>>>>>> DMPlexGetPartitioner(lDMBete, &lPart); >>>>>>>>> PetscPartitionerSetFromOptions(lPart); >>>>>>>>> >>>>>>>>> PetscSection section; >>>>>>>>> PetscInt numFields = 1; >>>>>>>>> PetscInt numBC = 0; >>>>>>>>> PetscInt numComp[1] = {1}; >>>>>>>>> PetscInt numDof[4] = {1, 0, 0, 0}; >>>>>>>>> PetscInt bcFields[1] = {0}; >>>>>>>>> IS bcPoints[1] = {NULL}; >>>>>>>>> >>>>>>>>> DMSetNumFields(lDMBete, numFields); >>>>>>>>> >>>>>>>>> DMPlexCreateSection(lDMBete, NULL, numComp, numDof, numBC, >>>>>>>>> bcFields, bcPoints, NULL, NULL, §ion); >>>>>>>>> DMSetLocalSection(lDMBete, section); >>>>>>>>> >>>>>>>>> DMPlexDistribute(lDMBete, 0, &lSFMigrationSansOvl, >>>>>>>>> &lDMDistribueSansOvl); // segfault! >>>>>>>>> >>>>>>>>> =========== >>>>>>>>> >>>>>>>>> So we have other question/remarks: >>>>>>>>> >>>>>>>>> 3- Maybe PETSc expect something specific that is missing/not >>>>>>>>> verified: for example, we didn't gave any coordinates since we just want to >>>>>>>>> partition and compute overlap for the mesh... and then recover our element >>>>>>>>> numbers in a "simple way" >>>>>>>>> >>>>>>>>> 4- We are telling ourselves it is somewhat a "big price to pay" to >>>>>>>>> have to build an unused section to have the global to natural ordering set >>>>>>>>> ? Could this requirement be avoided? >>>>>>>>> >>>>>>>> I don't think so. There would have to be _some_ way of describing >>>>>>>> your data layout in terms of mesh points, and I do not see how you could >>>>>>>> use less memory doing that. >>>>>>>> >>>>>>>>> 5- Are there any improvement towards our usages in 3.16 release? >>>>>>>>> >>>>>>>> Let me try and run the code above. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2021-09-29 7:39 p.m., Matthew Knepley wrote: >>>>>>>>> >>>>>>>>> On Wed, Sep 29, 2021 at 5:18 PM Eric Chamberland < >>>>>>>>> Eric.Chamberland at giref.ulaval.ca> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I come back with _almost_ the original question: >>>>>>>>>> >>>>>>>>>> I would like to add an integer information (*our* original >>>>>>>>>> element >>>>>>>>>> number, not petsc one) on each element of the DMPlex I create >>>>>>>>>> with >>>>>>>>>> DMPlexBuildFromCellListParallel. >>>>>>>>>> >>>>>>>>>> I would like this interger to be distribruted by or the same way >>>>>>>>>> DMPlexDistribute distribute the mesh. >>>>>>>>>> >>>>>>>>>> Is it possible to do this? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think we already have support for what you want. If you call >>>>>>>>> >>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMSetUseNatural.html >>>>>>>>> >>>>>>>>> before DMPlexDistribute(), it will compute a PetscSF encoding the >>>>>>>>> global to natural map. You >>>>>>>>> can get it with >>>>>>>>> >>>>>>>>> >>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGetGlobalToNaturalSF.html >>>>>>>>> >>>>>>>>> and use it with >>>>>>>>> >>>>>>>>> >>>>>>>>> https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexGlobalToNaturalBegin.html >>>>>>>>> >>>>>>>>> Is this sufficient? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> On 2021-07-14 1:18 p.m., Eric Chamberland wrote: >>>>>>>>>> > Hi, >>>>>>>>>> > >>>>>>>>>> > I want to use DMPlexDistribute from PETSc for computing >>>>>>>>>> overlapping >>>>>>>>>> > and play with the different partitioners supported. >>>>>>>>>> > >>>>>>>>>> > However, after calling DMPlexDistribute, I noticed the elements >>>>>>>>>> are >>>>>>>>>> > renumbered and then the original number is lost. >>>>>>>>>> > >>>>>>>>>> > What would be the best way to keep track of the element >>>>>>>>>> renumbering? >>>>>>>>>> > >>>>>>>>>> > a) Adding an optional parameter to let the user retrieve a >>>>>>>>>> vector or >>>>>>>>>> > "IS" giving the old number? >>>>>>>>>> > >>>>>>>>>> > b) Adding a DMLabel (seems a wrong good solution) >>>>>>>>>> > >>>>>>>>>> > c) Other idea? >>>>>>>>>> > >>>>>>>>>> > Of course, I don't want to loose performances with the need of >>>>>>>>>> this >>>>>>>>>> > "mapping"... >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > >>>>>>>>>> > Eric >>>>>>>>>> > >>>>>>>>>> -- >>>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>>> Professionnel de recherche >>>>>>>>>> GIREF/Universit? Laval >>>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Universit? Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>> Professionnel de recherche >>>>>>>> GIREF/Universit? Laval >>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Eric Chamberland, ing., M. Ing >>>>>>> Professionnel de recherche >>>>>>> GIREF/Universit? Laval >>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> >>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Universit? Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Universit? Laval >>>>> (418) 656-2131 poste 41 22 42 >>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Universit? Laval >>>> (418) 656-2131 poste 41 22 42 >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Universit? Laval >>> (418) 656-2131 poste 41 22 42 >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Universit? Laval >> (418) 656-2131 poste 41 22 42 >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Universit? Laval > (418) 656-2131 poste 41 22 42 > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbnbhlbilhmjdpfg.png Type: image/png Size: 42972 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: eejjfmbjimlkboec.png Type: image/png Size: 87901 bytes Desc: not available URL: From Eric.Chamberland at giref.ulaval.ca Wed Nov 10 22:44:59 2021 From: Eric.Chamberland at giref.ulaval.ca (Eric Chamberland) Date: Wed, 10 Nov 2021 23:44:59 -0500 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> <5751c2d4-e5d7-54d5-7f77-5bdb13c4a294@giref.ulaval.ca> Message-ID: <797cccf8-9a90-505c-aa3d-eac07877d64d@giref.ulaval.ca> Hi Matthew, I have joined the ex44.c example so you can see that we see. The problem is about the "f" field number which I can't see any clue in the documentation to what to put there... We are missing some information maybe written elsewhere? I have tried 3 different calls (lines 180-182) with comments aside: ? //ierr = DMSetBasicAdjacency(ddm, PETSC_TRUE, PETSC_FALSE); CHKERRQ(ierr); //This works: it give only 2 cells for the overlap ? //ierr = DMSetAdjacency(ddm, 0, PETSC_TRUE, PETSC_FALSE); CHKERRQ(ierr); //This works too, but why do I have to use field #0 ???? Is it a convention? ? ierr = DMSetAdjacency(ddm, PETSC_DEFAULT, PETSC_TRUE, PETSC_FALSE); CHKERRQ(ierr); //This does not works: it give 3 cells for the overlap instead of 2. Also, do you think your gitlab/knepley/fix-plex-g2n branch will find it's way into next or main? ;) Thanks, Eric On 2021-11-10 3:26 p.m., Matthew Knepley wrote: > Okay, so the PETSC_DEFAULT one is what is used for topology. There? > are two separate uses for adjacency info. > First, for this kind of topological extension, and the second is for > calculating Jacobians. You should be able to see > the difference between FEM and FVM in just the case of a 2x2 mesh > where each process gets 1 cell. With FEM, > the overlap is the whole mesh, whereas with FVM it leaves out the > diagonal partner. > > ? Thanks, > > ? ? ?Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ex44.c Type: text/x-csrc Size: 9055 bytes Desc: not available URL: From maj.rasouli at gmail.com Thu Nov 11 01:59:57 2021 From: maj.rasouli at gmail.com (Majid Rasouli) Date: Wed, 10 Nov 2021 23:59:57 -0800 Subject: [petsc-users] matrix-vector and matrix-matrix products in PETSc Message-ID: Dear all, I'm trying to find a brief explanation about the parallel (MPI) implementation of matrix-vector multiplication and matrix-matrix multiplication in PETSc. I have used them for some experiments for a research paper and I need to provide a brief explanation about the implementation of them. Is there a paper that I can read about that? If not, can anyone please help me find the source code that includes the implementation of those two functions? Thanks a lot, Majid. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Vladislav.Pimanov at skoltech.ru Thu Nov 11 03:39:47 2021 From: Vladislav.Pimanov at skoltech.ru (Vladislav Pimanov) Date: Thu, 11 Nov 2021 09:39:47 +0000 Subject: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. In-Reply-To: <0610114B-D10E-4A1F-BA31-9BBF067B6F22@petsc.dev> References: <6553e88affc34cf28cc8cda4f71a22a4@skoltech.ru> <0aea722c43a34006b03dd0eecde39d81@skoltech.ru>, <0610114B-D10E-4A1F-BA31-9BBF067B6F22@petsc.dev> Message-ID: <3340a5056fa543bf97b27450e1fc7d7e@skoltech.ru> Dear Barry, Let S = B A^{-1} B^T be the Schur complement, and \hat{S} = B diag(A)^{-1} B^T denotes the preconditioner. I also tried KSPComputeExtremeSingularValues for rtol(A) = rtol(\hat{S}) = 1e-14 as you suggested. However, the results did not change much compared to rtol(A) = rtol(S) = rtol(\hat{S}). Additionally, I checked the identity preconditioner instead of \hat{S}. In this case, the method firstly seems to converge when reaching rtol(S)=1e-6, but then, with a further decrease of rtol(S), it begins to diverge slightly though. I will try generalised EVP. Thanks, Vlad ________________________________ ??: Barry Smith ??????????: 10 ?????? 2021 ?. 17:45:55 ????: Vladislav Pimanov ?????: petsc-users at mcs.anl.gov ????: Re: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. P is a diffusion matrix, which itself is inverted by KSPCG) This worries me. Unless solved to full precision the action of solving with CG is not a linear operator in the input variable b, this means that the action of your Schur complement is not a linear operator and so iterative eigenvalue algorithms may not work correctly (even if the linear system seems to be converging). I suggest you try a KSP tolerance of 10^-14 for the inner KSP solve when you attempt the eigenvalue computation. Barry On Nov 10, 2021, at 9:09 AM, Vladislav Pimanov > wrote: Great thanks, Matt! The second option is what I was looking for. Best Regards, Vlad ________________________________ ??: Matthew Knepley > ??????????: 10 ?????? 2021 ?. 16:45:00 ????: Vladislav Pimanov ?????: petsc-users at mcs.anl.gov ????: Re: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. On Wed, Nov 10, 2021 at 8:42 AM Vladislav Pimanov > wrote: Dear PETSc community, I wonder if you could give me a hint on how to compute the condition number of a preconditioned matrix in a proper way. I have a MatSchurComplement matrix S and a preconditioner P of the type PCSHELL (P is a diffusion matrix, which itself is inverted by KSPCG). I tried to compute the condition number of P^{-1}S "for free" during the outer PCG procedure using KSPComputeExtremeSingularValues() routine. Unfortunately, \sigma_min does not converge even if the solution is computed with very high precision. I also looked at SLEPc interface, but did not realised how PC should be included. You can do this at least two ways: 1) Make a MatShell for P^{-1} S. This is easy, but you will not be able to use any factorization-type PC on that matrix. 2) Solve instead the generalized EVP, S x = \lambda P x. Since you already have P^{-1}, this should work well. Thanks, Matt Thanks! Sincerely, Vladislav Pimanov -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Nov 11 06:41:27 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Nov 2021 07:41:27 -0500 Subject: [petsc-users] matrix-vector and matrix-matrix products in PETSc In-Reply-To: References: Message-ID: On Thu, Nov 11, 2021 at 3:00 AM Majid Rasouli wrote: > Dear all, > > I'm trying to find a brief explanation about the parallel (MPI) > implementation of matrix-vector multiplication and matrix-matrix > multiplication in PETSc. I have used them for some experiments for a > research paper and I need to provide a brief explanation about the > implementation of them. Is there a paper that I can read about that? If > not, can anyone please help me find the source code that includes the > implementation of those two functions? > I think this describes our SpMV: https://dl.acm.org/doi/abs/10.1145/3225058.3225100?casa_token=Pz_zMwXCoz8AAAAA%3AH8iNvXSY4opzDsJsEnoYVU0CccQjOK9pIH4Rs_Yb7a_ebchRpgAJ5GHoH2humWfkFYS54m2fO84s This paper describes the SpMM algorithms we use, and makes some notes on performance https://epubs.siam.org/doi/abs/10.1137/15M1028807 but I could not find a paper on our implementation. Thanks, Matt > Thanks a lot, > Majid. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Nov 11 06:59:25 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Nov 2021 07:59:25 -0500 Subject: [petsc-users] Is it possible to keep track of original elements # after a call to DMPlexDistribute ? In-Reply-To: <797cccf8-9a90-505c-aa3d-eac07877d64d@giref.ulaval.ca> References: <7236c736-6066-1ba3-55b1-60782d8e754f@giref.ulaval.ca> <12e32ebb-61ed-6a8c-ab77-2841090ba5fe@giref.ulaval.ca> <8a3704c3-d626-d860-0e98-33e113c5c376@giref.ulaval.ca> <342410e9-f3f4-21de-51c4-09bc273281ca@giref.ulaval.ca> <5751c2d4-e5d7-54d5-7f77-5bdb13c4a294@giref.ulaval.ca> <797cccf8-9a90-505c-aa3d-eac07877d64d@giref.ulaval.ca> Message-ID: On Wed, Nov 10, 2021 at 11:45 PM Eric Chamberland < Eric.Chamberland at giref.ulaval.ca> wrote: > Hi Matthew, > > I have joined the ex44.c example so you can see that we see. > > The problem is about the "f" field number which I can't see any clue in > the documentation to what to put there... We are missing some information > maybe written elsewhere? > > I have tried 3 different calls (lines 180-182) with comments aside: > > //ierr = DMSetBasicAdjacency(ddm, PETSC_TRUE, PETSC_FALSE); > CHKERRQ(ierr); //This works: it give only 2 cells for the overlap //ierr > = DMSetAdjacency(ddm, 0, PETSC_TRUE, PETSC_FALSE); CHKERRQ(ierr); //This > works too, but why do I have to use field #0 ??? Is it a convention? > ierr = DMSetAdjacency(ddm, PETSC_DEFAULT, PETSC_TRUE, PETSC_FALSE); > CHKERRQ(ierr); //This does not works: it give 3 cells for the overlap > instead of 2. > Okay, I will fix the docs. I will run it, but I think I see what is happening here. The default is used _only_ when no fields are defined. This is to support the case of just using Plex without any field definitions. Once you define fields, we let you have a different adjacency for each field, so that Jacobian layout can be customized. However, what is used for topology then? Right now, I believe it is field 0. It sounds like I need a flag to tell me whether the user set something, so we can have better defaults. > Also, do you think your gitlab/knepley/fix-plex-g2n branch will find it's > way into next or main? ;) > Put in the MR today. Thanks, Matt > Thanks, > > Eric > > On 2021-11-10 3:26 p.m., Matthew Knepley wrote: > > Okay, so the PETSC_DEFAULT one is what is used for topology. There are > two separate uses for adjacency info. > First, for this kind of topological extension, and the second is for > calculating Jacobians. You should be able to see > the difference between FEM and FVM in just the case of a 2x2 mesh where > each process gets 1 cell. With FEM, > the overlap is the whole mesh, whereas with FVM it leaves out the diagonal > partner. > > Thanks, > > Matt > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Thu Nov 11 07:33:19 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Thu, 11 Nov 2021 14:33:19 +0100 Subject: [petsc-users] matrix-vector and matrix-matrix products in PETSc In-Reply-To: References: Message-ID: <83C83A03-38FC-44D6-981E-BAC6490FC354@dsic.upv.es> > El 11 nov 2021, a las 13:41, Matthew Knepley escribi?: > > On Thu, Nov 11, 2021 at 3:00 AM Majid Rasouli wrote: > Dear all, > > I'm trying to find a brief explanation about the parallel (MPI) implementation of matrix-vector multiplication and matrix-matrix multiplication in PETSc. I have used them for some experiments for a research paper and I need to provide a brief explanation about the implementation of them. Is there a paper that I can read about that? If not, can anyone please help me find the source code that includes the implementation of those two functions? > > I think this describes our SpMV: > > https://dl.acm.org/doi/abs/10.1145/3225058.3225100?casa_token=Pz_zMwXCoz8AAAAA%3AH8iNvXSY4opzDsJsEnoYVU0CccQjOK9pIH4Rs_Yb7a_ebchRpgAJ5GHoH2humWfkFYS54m2fO84s > > This paper describes the SpMM algorithms we use, and makes some notes on performance > > https://epubs.siam.org/doi/abs/10.1137/15M1028807 > > but I could not find a paper on our implementation. Sparse Matrix-Matrix Products Executed Through Coloring Michael McCourt, Barry Smith, and Hong Zhang https://doi.org/10.1137/13093426X > > Thanks, > > Matt > > Thanks a lot, > Majid. > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ From bsmith at petsc.dev Thu Nov 11 09:14:26 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 11 Nov 2021 10:14:26 -0500 Subject: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. In-Reply-To: <3340a5056fa543bf97b27450e1fc7d7e@skoltech.ru> References: <6553e88affc34cf28cc8cda4f71a22a4@skoltech.ru> <0aea722c43a34006b03dd0eecde39d81@skoltech.ru> <0610114B-D10E-4A1F-BA31-9BBF067B6F22@petsc.dev> <3340a5056fa543bf97b27450e1fc7d7e@skoltech.ru> Message-ID: I think there is likely a problem somewhere along the line in the libraries or how you are using them that causes the problem. You can run with a small problem and explicit form the inner operators and use direct solvers for the inner inverses to see what happens to the convergence of the computation of the eigenvalues. You can also use -ksp_view_eigenvalues_explicit and things like -ksp_view_mat_explicit and -ksp_view_preconditioned_operator_explicit to dump representations of the operators (for small problems) and load them in Matlab to get the eigenvalues and see if they match what the PETSc eigenvalue computation gives. > On Nov 11, 2021, at 4:39 AM, Vladislav Pimanov wrote: > > Dear Barry, > > Let S = B A^{-1} B^T be the Schur complement, and \hat{S} = B diag(A)^{-1} B^T denotes the preconditioner. > I also tried KSPComputeExtremeSingularValues for rtol(A) = rtol(\hat{S}) = 1e-14 as you suggested. However, the results did not change much compared to rtol(A) = rtol(S) = rtol(\hat{S}). > > Additionally, I checked the identity preconditioner instead of \hat{S}. In this case, the method firstly seems to converge when reaching rtol(S)=1e-6, but then, with a further decrease of rtol(S), it begins to diverge slightly though. > > I will try generalised EVP. > > Thanks, > Vlad > > ??: Barry Smith > > ??????????: 10 ?????? 2021 ?. 17:45:55 > ????: Vladislav Pimanov > ?????: petsc-users at mcs.anl.gov > ????: Re: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. > > >> P is a diffusion matrix, which itself is inverted by KSPCG) > > This worries me. Unless solved to full precision the action of solving with CG is not a linear operator in the input variable b, this means that the action of your Schur complement is not a linear operator and so iterative eigenvalue algorithms may not work correctly (even if the linear system seems to be converging). > > I suggest you try a KSP tolerance of 10^-14 for the inner KSP solve when you attempt the eigenvalue computation. > > Barry > > >> On Nov 10, 2021, at 9:09 AM, Vladislav Pimanov > wrote: >> >> Great thanks, Matt! >> The second option is what I was looking for. >> >> Best Regards, >> Vlad >> ??: Matthew Knepley > >> ??????????: 10 ?????? 2021 ?. 16:45:00 >> ????: Vladislav Pimanov >> ?????: petsc-users at mcs.anl.gov >> ????: Re: [petsc-users] How to compute the condition number of SchurComplementMat preconditioned with PCSHELL. >> >> On Wed, Nov 10, 2021 at 8:42 AM Vladislav Pimanov > wrote: >> Dear PETSc community, >> >> >> I wonder if you could give me a hint on how to compute the condition number of a preconditioned matrix in a proper way. >> I have a MatSchurComplement matrix S and a preconditioner P of the type PCSHELL (P is a diffusion matrix, which itself is inverted by KSPCG). >> I tried to compute the condition number of P^{-1}S "for free" during the outer PCG procedure using KSPComputeExtremeSingularValues() routine. >> Unfortunately, \sigma_min does not converge even if the solution is computed with very high precision. >> I also looked at SLEPc interface, but did not realised how PC should be included. >> >> You can do this at least two ways: >> >> 1) Make a MatShell for P^{-1} S. This is easy, but you will not be able to use any factorization-type PC on that matrix. >> >> 2) Solve instead the generalized EVP, S x = \lambda P x. Since you already have P^{-1}, this should work well. >> >> Thanks, >> >> Matt >> Thanks! >> >> Sincerely, >> Vladislav Pimanov >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hzhang at mcs.anl.gov Thu Nov 11 10:15:01 2021 From: hzhang at mcs.anl.gov (Zhang, Hong) Date: Thu, 11 Nov 2021 16:15:01 +0000 Subject: [petsc-users] matrix-vector and matrix-matrix products in PETSc In-Reply-To: <83C83A03-38FC-44D6-981E-BAC6490FC354@dsic.upv.es> References: <83C83A03-38FC-44D6-981E-BAC6490FC354@dsic.upv.es> Message-ID: "Sparse Matrix-Matrix Products Executed Through Coloring" paper is a sequential algorithm for C=A*B^T. We do not have paper discussing the parallel mat-mat product algorithms used in PETSc. For source code, you may look at MatMatMultSymbolic_MPIAIJ_MPIAIJ() and MatMatMultNumeric_MPIAIJ_MPIAIJ() in petsc/src/mat/impls/aij/mpi/mpimatmatmult.c. Hong ________________________________ From: petsc-users on behalf of Jose E. Roman Sent: Thursday, November 11, 2021 7:33 AM To: Matthew Knepley Cc: PETSc ; Majid Rasouli Subject: Re: [petsc-users] matrix-vector and matrix-matrix products in PETSc > El 11 nov 2021, a las 13:41, Matthew Knepley escribi?: > > On Thu, Nov 11, 2021 at 3:00 AM Majid Rasouli wrote: > Dear all, > > I'm trying to find a brief explanation about the parallel (MPI) implementation of matrix-vector multiplication and matrix-matrix multiplication in PETSc. I have used them for some experiments for a research paper and I need to provide a brief explanation about the implementation of them. Is there a paper that I can read about that? If not, can anyone please help me find the source code that includes the implementation of those two functions? > > I think this describes our SpMV: > > https://dl.acm.org/doi/abs/10.1145/3225058.3225100?casa_token=Pz_zMwXCoz8AAAAA%3AH8iNvXSY4opzDsJsEnoYVU0CccQjOK9pIH4Rs_Yb7a_ebchRpgAJ5GHoH2humWfkFYS54m2fO84s > > This paper describes the SpMM algorithms we use, and makes some notes on performance > > https://epubs.siam.org/doi/abs/10.1137/15M1028807 > > but I could not find a paper on our implementation. Sparse Matrix-Matrix Products Executed Through Coloring Michael McCourt, Barry Smith, and Hong Zhang https://doi.org/10.1137/13093426X > > Thanks, > > Matt > > Thanks a lot, > Majid. > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhargav.subramanya at kaust.edu.sa Thu Nov 11 12:17:21 2021 From: bhargav.subramanya at kaust.edu.sa (Bhargav Subramanya) Date: Thu, 11 Nov 2021 21:17:21 +0300 Subject: [petsc-users] dmplex extrude in parallel In-Reply-To: References: Message-ID: Dear Matt, I just realized that I used PETSC_COMM_SELF instead of PETSC_COMM_WORLD while performing the above test. I am sorry for that. After fixing it, I have the following 3 cases: 1. With the below command line options, where I simply create the spherical surface mesh, everything is fine. The mesh gets distributed in parallel with the check DMPlexIsDistributed. /home/subrambm/petsc/arch-linux-c-debug/bin/mpiexec -n 2 ./cavity_flow.out -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_shape sphere -dm_plex_sphere_radius 1.0 -dm_refine_pre 2 -dm_view vtk:mesh.vtk -dm_distribute 1 ------------------------------------------------------------------- 2. This case also seems to work. However, the mesh file data is bad. Do you think it's because of the vtk file format? -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_extrude_layers 4 -dm_extrude_thickness 0.1 -dm_extrude_column_first 0 -dm_distribute 1 -dm_view vtk:mesh.vtk ------------------------------------------------------------------- 3. This is the case which I want: To distribute the mesh, refine it in parallel and extrude it. However, after distribution, the mesh does not seem to be refined as I get the following error: -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_distribute 1 -dm_refine 1 Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). Ignoring PCI device with non-16bit domain. Pass --enable-32bits-pci-domain to configure to support such devices (warning: it would break the library ABI, don't enable unless really needed). [1]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [1]PETSC ERROR: Nonconforming object sizes [1]PETSC ERROR: The output section point (0) closure size 9 != dual space dimension 12 at height 0 in [0, 0] [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [1]PETSC ERROR: Petsc Release Version 3.16.1, unknown [1]PETSC ERROR: ./cavity_flow.out on a arch-linux-c-debug named kw60970 by subrambm Thu Nov 11 21:13:27 2021 [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-superlu_dist --download-hypre --download-fiat --download-generator --download-triangle --download-tetgen --download-chaco --download-make -download-boost --download-cmake --download-ml --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-hdf5 [1]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() at /home/subrambm/petsc/src/dm/impls/plex/plexproject.c:762 [1]PETSC ERROR: #2 DMProjectFieldLocal_Plex() at /home/subrambm/petsc/src/dm/impls/plex/plexproject.c:933 [1]PETSC ERROR: #3 DMProjectFieldLocal() at /home/subrambm/petsc/src/dm/interface/dm.c:8863 [1]PETSC ERROR: #4 DMPlexRemapGeometry() at /home/subrambm/petsc/src/dm/impls/plex/plexgeometry.c:3270 [1]PETSC ERROR: #5 DMSetFromOptions_Plex() at /home/subrambm/petsc/src/dm/impls/plex/plexcreate.c:3064 [1]PETSC ERROR: #6 DMSetFromOptions() at /home/subrambm/petsc/src/dm/interface/dm.c:902 [1]PETSC ERROR: #7 main() at meshtest.c:12 [1]PETSC ERROR: PETSc Option Table entries: [1]PETSC ERROR: -dm_distribute 1 [1]PETSC ERROR: -dm_plex_dim 2 [1]PETSC ERROR: -dm_plex_shape sphere [1]PETSC ERROR: -dm_plex_simplex 1 [1]PETSC ERROR: -dm_plex_sphere_radius 1.0 [1]PETSC ERROR: -dm_refine 1 [1]PETSC ERROR: -dm_refine_pre 3 [1]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1 Thanks, Bhargav On Wed, Nov 10, 2021 at 4:33 PM Matthew Knepley wrote: > On Wed, Nov 10, 2021 at 8:26 AM Bhargav Subramanya < > bhargav.subramanya at kaust.edu.sa> wrote: > >> Dear Matt, >> >> Thanks a lot for the reply. I am now able to generate the prismatic mesh >> properly. >> > > Cool. > > >> In the case of mpiexec -n 2 ./meshtest -dm_plex_shape sphere >> -dm_refine_pre 3 -dm_distribute -dm_refine 2 and DMExtrude(), where I am >> ordering the extruded cells on the layers first; Is the mesh extruded in >> parallel for this case? >> > > Yes. > > >> The first time I distributed the mesh, it was only the spherical surface >> mesh. After dmplex extrude, the mesh changes. Do I need to redistribute the >> mesh again? >> > > Probably not. The balance factor will not change, meaning the relative > sizes of the partitions. However, if you have a lot of layers, maybe you > want to. It is easy, > just call DMDistribute() again. > > Thanks, > > Matt > > >> Thanks, >> Bhargav >> >> On Wed, Nov 10, 2021 at 12:56 AM Matthew Knepley >> wrote: >> >>> On Tue, Nov 9, 2021 at 9:54 AM Bhargav Subramanya < >>> bhargav.subramanya at kaust.edu.sa> wrote: >>> >>>> Dear All, >>>> >>>> I want to generate a prismatic mesh from a base spherical surface mesh. >>>> I first refine the base icosahedron sequential mesh to some extent, >>>> distribute the mesh, continue the refinement in parallel, project the mesh >>>> to a unit sphere, and then finally extrude the mesh. The following is the >>>> code I am using. As the extrusion is performed in parallel, the extruded >>>> mesh seems to be broken as shown in the attached figure. May I know how to >>>> get an intact extruded mesh in parallel? Also, is it possible to make mesh >>>> refinement respect the spherical surface geometry, without having to >>>> project it using a function as shown below? >>>> >>> >>> I think you can do most of what you want from the command line. Here is >>> a simple code: >>> >>> #include >>> >>> int main(int argc, char **argv) >>> { >>> DM dm; >>> PetscErrorCode ierr; >>> >>> ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; >>> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); >>> ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); >>> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >>> ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); >>> ierr = DMDestroy(&dm);CHKERRQ(ierr); >>> ierr = PetscFinalize(); >>> return ierr; >>> } >>> >>> that I run using >>> >>> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 >>> -dm_extrude 5 -dm_plex_transform_extrude_thickness 0.1 -dm_distribute >>> -dm_view hdf5:mesh.h5 >>> >>> and get the attached picture. I am refining before distribution only >>> here, since the after distribution refinement would refine the extrude >>> thing, which is not what you want. >>> If you want refinement after distribution, you could use this code to do >>> >>> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre >>> 3 -dm_distribute -dm_refine 2 >>> >>> and then call DMExtrude() by hand after that. Alternatively, I could >>> make a pre-extrusion refinement option. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Bhargav >>>> >>>> /* refine the sequential mesh first */ >>>> for (int r = 0; r < numRefinePre; ++r) { >>>> DM rdm = NULL; >>>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>>> DMDestroy(&dm); >>>> dm = rdm; >>>> } >>>> >>>> /* project to a unit sphere */ >>>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>>> >>>> /* create and distribute DM */ >>>> ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); >>>> if (dmDist) { >>>> DMDestroy(&dm); >>>> dm = dmDist; >>>> } >>>> >>>> /* refine the mesh in parallel */ >>>> for (int r = 0; r < numRefine; ++r) { >>>> DM rdm = NULL; >>>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>>> DMDestroy(&dm); >>>> dm = rdm; >>>> } >>>> >>>> /* project to a unit sphere */ >>>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>>> >>>> ierr = DMPlexExtrude(dm, numLayers-1, radialThickness, PETSC_FALSE, >>>> NULL, PETSC_TRUE, &dm3D);CHKERRQ(ierr); >>>> >>>> if (dm3D) { >>>> DMDestroy(&dm); >>>> dm = dm3D; >>>> } >>>> >>>> >>>> ------------------------------ >>>> This message and its contents, including attachments are intended >>>> solely for the original recipient. If you are not the intended recipient or >>>> have received this message in error, please notify me immediately and >>>> delete this message from your computer system. Any unauthorized use or >>>> distribution is prohibited. Please consider the environment before printing >>>> this email. >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> ------------------------------ >> This message and its contents, including attachments are intended solely >> for the original recipient. If you are not the intended recipient or have >> received this message in error, please notify me immediately and delete >> this message from your computer system. Any unauthorized use or >> distribution is prohibited. Please consider the environment before printing >> this email. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Nov 11 12:41:27 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Nov 2021 13:41:27 -0500 Subject: [petsc-users] dmplex extrude in parallel In-Reply-To: References: Message-ID: On Thu, Nov 11, 2021 at 1:17 PM Bhargav Subramanya < bhargav.subramanya at kaust.edu.sa> wrote: > Dear Matt, > > I just realized that I used PETSC_COMM_SELF instead of PETSC_COMM_WORLD > while performing the above test. I am sorry for that. After fixing it, I > have the following 3 cases: > > 1. With the below command line options, where I simply create the > spherical surface mesh, everything is fine. The mesh gets distributed in > parallel with the check DMPlexIsDistributed. > /home/subrambm/petsc/arch-linux-c-debug/bin/mpiexec -n 2 ./cavity_flow.out > -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_shape sphere > -dm_plex_sphere_radius 1.0 -dm_refine_pre 2 -dm_view vtk:mesh.vtk > -dm_distribute 1 > > ------------------------------------------------------------------- > 2. This case also seems to work. However, the mesh file data is bad. Do > you think it's because of the vtk file format? > -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 > -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_extrude_layers 4 > -dm_extrude_thickness 0.1 -dm_extrude_column_first 0 -dm_distribute 1 > -dm_view vtk:mesh.vtk > It should work, but that would be the first place I would check for a bug. I would use -dm_view hdf5:mesh.h5 and then $PETSC_DIR/lib/petsc/bin/petsc_gen_xdmf.py mesh.h5 to make mesh.xmf, which can be loaded by ParaView. > ------------------------------------------------------------------- > > 3. This is the case which I want: To distribute the mesh, refine it in > parallel and extrude it. > However, after distribution, the mesh does not seem to be refined as I get > the following error: > -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 > -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_distribute 1 -dm_refine 1 > The problem here is that you are refining the extruded mesh, not refining the surface mesh. There must be something switched around in your file. Thanks, Matt > Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > Ignoring PCI device with non-16bit domain. > Pass --enable-32bits-pci-domain to configure to support such devices > (warning: it would break the library ABI, don't enable unless really > needed). > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Nonconforming object sizes > [1]PETSC ERROR: The output section point (0) closure size 9 != dual space > dimension 12 at height 0 in [0, 0] > [1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [1]PETSC ERROR: Petsc Release Version 3.16.1, unknown > [1]PETSC ERROR: ./cavity_flow.out on a arch-linux-c-debug named kw60970 by > subrambm Thu Nov 11 21:13:27 2021 > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > --download-superlu_dist --download-hypre --download-fiat > --download-generator --download-triangle --download-tetgen --download-chaco > --download-make -download-boost --download-cmake --download-ml > --download-mumps --download-scalapack --download-parmetis --download-metis > --download-ptscotch --download-hdf5 > [1]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() at > /home/subrambm/petsc/src/dm/impls/plex/plexproject.c:762 > [1]PETSC ERROR: #2 DMProjectFieldLocal_Plex() at > /home/subrambm/petsc/src/dm/impls/plex/plexproject.c:933 > [1]PETSC ERROR: #3 DMProjectFieldLocal() at > /home/subrambm/petsc/src/dm/interface/dm.c:8863 > [1]PETSC ERROR: #4 DMPlexRemapGeometry() at > /home/subrambm/petsc/src/dm/impls/plex/plexgeometry.c:3270 > [1]PETSC ERROR: #5 DMSetFromOptions_Plex() at > /home/subrambm/petsc/src/dm/impls/plex/plexcreate.c:3064 > [1]PETSC ERROR: #6 DMSetFromOptions() at > /home/subrambm/petsc/src/dm/interface/dm.c:902 > [1]PETSC ERROR: #7 main() at meshtest.c:12 > [1]PETSC ERROR: PETSc Option Table entries: > [1]PETSC ERROR: -dm_distribute 1 > [1]PETSC ERROR: -dm_plex_dim 2 > [1]PETSC ERROR: -dm_plex_shape sphere > [1]PETSC ERROR: -dm_plex_simplex 1 > [1]PETSC ERROR: -dm_plex_sphere_radius 1.0 > [1]PETSC ERROR: -dm_refine 1 > [1]PETSC ERROR: -dm_refine_pre 3 > [1]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1 > > Thanks, > Bhargav > > On Wed, Nov 10, 2021 at 4:33 PM Matthew Knepley wrote: > >> On Wed, Nov 10, 2021 at 8:26 AM Bhargav Subramanya < >> bhargav.subramanya at kaust.edu.sa> wrote: >> >>> Dear Matt, >>> >>> Thanks a lot for the reply. I am now able to generate the prismatic mesh >>> properly. >>> >> >> Cool. >> >> >>> In the case of mpiexec -n 2 ./meshtest -dm_plex_shape sphere >>> -dm_refine_pre 3 -dm_distribute -dm_refine 2 and DMExtrude(), where I am >>> ordering the extruded cells on the layers first; Is the mesh extruded in >>> parallel for this case? >>> >> >> Yes. >> >> >>> The first time I distributed the mesh, it was only the spherical surface >>> mesh. After dmplex extrude, the mesh changes. Do I need to redistribute the >>> mesh again? >>> >> >> Probably not. The balance factor will not change, meaning the relative >> sizes of the partitions. However, if you have a lot of layers, maybe you >> want to. It is easy, >> just call DMDistribute() again. >> >> Thanks, >> >> Matt >> >> >>> Thanks, >>> Bhargav >>> >>> On Wed, Nov 10, 2021 at 12:56 AM Matthew Knepley >>> wrote: >>> >>>> On Tue, Nov 9, 2021 at 9:54 AM Bhargav Subramanya < >>>> bhargav.subramanya at kaust.edu.sa> wrote: >>>> >>>>> Dear All, >>>>> >>>>> I want to generate a prismatic mesh from a base spherical surface >>>>> mesh. I first refine the base icosahedron sequential mesh to some extent, >>>>> distribute the mesh, continue the refinement in parallel, project the mesh >>>>> to a unit sphere, and then finally extrude the mesh. The following is the >>>>> code I am using. As the extrusion is performed in parallel, the extruded >>>>> mesh seems to be broken as shown in the attached figure. May I know how to >>>>> get an intact extruded mesh in parallel? Also, is it possible to make mesh >>>>> refinement respect the spherical surface geometry, without having to >>>>> project it using a function as shown below? >>>>> >>>> >>>> I think you can do most of what you want from the command line. Here is >>>> a simple code: >>>> >>>> #include >>>> >>>> int main(int argc, char **argv) >>>> { >>>> DM dm; >>>> PetscErrorCode ierr; >>>> >>>> ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return >>>> ierr; >>>> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); >>>> ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); >>>> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >>>> ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); >>>> ierr = DMDestroy(&dm);CHKERRQ(ierr); >>>> ierr = PetscFinalize(); >>>> return ierr; >>>> } >>>> >>>> that I run using >>>> >>>> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 >>>> -dm_extrude 5 -dm_plex_transform_extrude_thickness 0.1 -dm_distribute >>>> -dm_view hdf5:mesh.h5 >>>> >>>> and get the attached picture. I am refining before distribution only >>>> here, since the after distribution refinement would refine the extrude >>>> thing, which is not what you want. >>>> If you want refinement after distribution, you could use this code to do >>>> >>>> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre >>>> 3 -dm_distribute -dm_refine 2 >>>> >>>> and then call DMExtrude() by hand after that. Alternatively, I could >>>> make a pre-extrusion refinement option. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> Thanks, >>>>> Bhargav >>>>> >>>>> /* refine the sequential mesh first */ >>>>> for (int r = 0; r < numRefinePre; ++r) { >>>>> DM rdm = NULL; >>>>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>>>> DMDestroy(&dm); >>>>> dm = rdm; >>>>> } >>>>> >>>>> /* project to a unit sphere */ >>>>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>>>> >>>>> /* create and distribute DM */ >>>>> ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); >>>>> if (dmDist) { >>>>> DMDestroy(&dm); >>>>> dm = dmDist; >>>>> } >>>>> >>>>> /* refine the mesh in parallel */ >>>>> for (int r = 0; r < numRefine; ++r) { >>>>> DM rdm = NULL; >>>>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>>>> DMDestroy(&dm); >>>>> dm = rdm; >>>>> } >>>>> >>>>> /* project to a unit sphere */ >>>>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>>>> >>>>> ierr = DMPlexExtrude(dm, numLayers-1, radialThickness, PETSC_FALSE, >>>>> NULL, PETSC_TRUE, &dm3D);CHKERRQ(ierr); >>>>> >>>>> if (dm3D) { >>>>> DMDestroy(&dm); >>>>> dm = dm3D; >>>>> } >>>>> >>>>> >>>>> ------------------------------ >>>>> This message and its contents, including attachments are intended >>>>> solely for the original recipient. If you are not the intended recipient or >>>>> have received this message in error, please notify me immediately and >>>>> delete this message from your computer system. Any unauthorized use or >>>>> distribution is prohibited. Please consider the environment before printing >>>>> this email. >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> >>>> >>> >>> ------------------------------ >>> This message and its contents, including attachments are intended solely >>> for the original recipient. If you are not the intended recipient or have >>> received this message in error, please notify me immediately and delete >>> this message from your computer system. Any unauthorized use or >>> distribution is prohibited. Please consider the environment before printing >>> this email. >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Thu Nov 11 14:23:45 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Thu, 11 Nov 2021 13:23:45 -0700 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: <61ccdb88-8636-cf84-69f5-eda0a1a6294b@mcs.anl.gov> References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> <61ccdb88-8636-cf84-69f5-eda0a1a6294b@mcs.anl.gov> Message-ID: Thanks, Satish "--with-make-np=1" did help us on MUMPS, but we had new trouble with hypre now. It is hard to understand why "gmake install" even failed. Please see the attachment for the log file. Thanks, Fande On Wed, Nov 10, 2021 at 12:16 PM Satish Balay wrote: > You are using petsc-3.15.1 - and likely the mumps build change between > then and current 3.16. > > Can you use latest PETSc release? > > If not - Suggest removing --download-mumps= > https://bitbucket.org/petsc/pkg-mumps.git > --download-mumps-commit=v5.4.1-p1 options [and PETSC_ARCH] and going back > to your old > build. > > If it fails [as before] - retry with: --with-make-np=1 > > Satish > > On Wed, 10 Nov 2021, Novak, April via petsc-users wrote: > > > Hi Barry, > > > > Thank you for your assistance - I?ve attached the latest configure.log. > I still encounter issues building, though some of the MUMPS errors do seem > to have been fixed with the --download-mumps-commit option. Do you have a > recommendation for addressing these other errors? > > > > Thanks, > > > > -April > > > > From: Fande Kong > > Date: Wednesday, November 10, 2021 at 9:44 AM > > To: Barry Smith > > Cc: PETSc users list , Novak, April < > anovak at anl.gov> > > Subject: Re: [petsc-users] Error running make on MUMPS > > Thanks, Barry, > > > > We will try this, and report back > > > > > > Fande > > > > On Tue, Nov 9, 2021 at 5:41 PM Barry Smith bsmith at petsc.dev>> wrote: > > > > This version of MUMPS has a bug in its build system; it does not have > all the dependencies on Fortran modules properly listed so Fortran files > can get compiled too early causing "random" failures during some builds, > especially on machines with lots of cores for compiling. > > > > I think you should be able to use --download-mumps= > https://bitbucket.org/petsc/pkg-mumps.git > --download-mumps-commit=v5.4.1-p1 to get a patched version. > > > > Barry > > > > > > > > On Nov 9, 2021, at 6:10 PM, Fande Kong fdkong.jd at gmail.com>> wrote: > > > > Hi All, > > > > We encountered a configuration error when running the PETSc > configuration on a HPC system. Went through the log file, but could not > find much. The log file was attached. > > > > Any thoughts? > > > > Thanks for your help, as always. > > > > Fande > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: application/octet-stream Size: 2954493 bytes Desc: not available URL: From knepley at gmail.com Thu Nov 11 14:28:58 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Nov 2021 15:28:58 -0500 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> <61ccdb88-8636-cf84-69f5-eda0a1a6294b@mcs.anl.gov> Message-ID: On Thu, Nov 11, 2021 at 3:25 PM Fande Kong wrote: > Thanks, Satish > > "--with-make-np=1" did help us on MUMPS, but we had new trouble with > hypre now. > > It is hard to understand why "gmake install" even failed. > Because HYPRE thinks it is better to use 'ln' than the 'install' script that handles things like the target existing: gmake[1]: Leaving directory `/beegfs1/home/anovak/cardinal/contrib/moose/petsc/arch-moose/externalpackages/git.hypre/src/parcsr_ls' ln: failed to create symbolic link ?libHYPRE_parcsr_ls.so?: File exists gmake[1]: *** [libHYPRE_parcsr_ls.so] Error 1 gmake: *** [all] Error 1 I think everything needs to be cleaned out for Hypre to reinstall. Thanks, Matt > Please see the attachment for the log file. > > Thanks, > > Fande > > On Wed, Nov 10, 2021 at 12:16 PM Satish Balay wrote: > >> You are using petsc-3.15.1 - and likely the mumps build change between >> then and current 3.16. >> >> Can you use latest PETSc release? >> >> If not - Suggest removing --download-mumps= >> https://bitbucket.org/petsc/pkg-mumps.git >> --download-mumps-commit=v5.4.1-p1 options [and PETSC_ARCH] and going back >> to your old >> build. >> >> If it fails [as before] - retry with: --with-make-np=1 >> >> Satish >> >> On Wed, 10 Nov 2021, Novak, April via petsc-users wrote: >> >> > Hi Barry, >> > >> > Thank you for your assistance - I?ve attached the latest configure.log. >> I still encounter issues building, though some of the MUMPS errors do seem >> to have been fixed with the --download-mumps-commit option. Do you have a >> recommendation for addressing these other errors? >> > >> > Thanks, >> > >> > -April >> > >> > From: Fande Kong >> > Date: Wednesday, November 10, 2021 at 9:44 AM >> > To: Barry Smith >> > Cc: PETSc users list , Novak, April < >> anovak at anl.gov> >> > Subject: Re: [petsc-users] Error running make on MUMPS >> > Thanks, Barry, >> > >> > We will try this, and report back >> > >> > >> > Fande >> > >> > On Tue, Nov 9, 2021 at 5:41 PM Barry Smith > bsmith at petsc.dev>> wrote: >> > >> > This version of MUMPS has a bug in its build system; it does not >> have all the dependencies on Fortran modules properly listed so Fortran >> files can get compiled too early causing "random" failures during some >> builds, especially on machines with lots of cores for compiling. >> > >> > I think you should be able to use --download-mumps= >> https://bitbucket.org/petsc/pkg-mumps.git >> --download-mumps-commit=v5.4.1-p1 to get a patched version. >> > >> > Barry >> > >> > >> > >> > On Nov 9, 2021, at 6:10 PM, Fande Kong > fdkong.jd at gmail.com>> wrote: >> > >> > Hi All, >> > >> > We encountered a configuration error when running the PETSc >> configuration on a HPC system. Went through the log file, but could not >> find much. The log file was attached. >> > >> > Any thoughts? >> > >> > Thanks for your help, as always. >> > >> > Fande >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Thu Nov 11 14:44:43 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Thu, 11 Nov 2021 13:44:43 -0700 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> <61ccdb88-8636-cf84-69f5-eda0a1a6294b@mcs.anl.gov> Message-ID: Thanks Matt, I understand completely, the actual error should be " ln -s libHYPRE_parcsr_ls-2.20.0.so libHYPRE_parcsr_ls.so gmake[1]: Leaving directory `/beegfs1/home/anovak/cardinal/contrib/moose/petsc/arch-moose/externalpackages/git.hypre/src/parcsr_ls' Error running make; make install on HYPRE: Could not execute "['/usr/bin/gmake install']": " The one you saw was that hypre automatically took the second try for parcsr_ls after the first failed gmake install. Because the first try already did "ln -s libHYPRE_parcsr_ls-2.20.0.so libHYPRE_parcsr_ls.so", and then second try would see that ?libHYPRE_parcsr_ls.so" already existed. Thanks, Fande On Thu, Nov 11, 2021 at 1:29 PM Matthew Knepley wrote: > On Thu, Nov 11, 2021 at 3:25 PM Fande Kong wrote: > >> Thanks, Satish >> >> "--with-make-np=1" did help us on MUMPS, but we had new trouble with >> hypre now. >> >> It is hard to understand why "gmake install" even failed. >> > > Because HYPRE thinks it is better to use 'ln' than the 'install' script > that handles things like the target existing: > > gmake[1]: Leaving directory > `/beegfs1/home/anovak/cardinal/contrib/moose/petsc/arch-moose/externalpackages/git.hypre/src/parcsr_ls' > ln: failed to create symbolic link ?libHYPRE_parcsr_ls.so?: File exists > gmake[1]: *** [libHYPRE_parcsr_ls.so] Error 1 > gmake: *** [all] Error 1 > > I think everything needs to be cleaned out for Hypre to reinstall. > > Thanks, > > Matt > > >> Please see the attachment for the log file. >> >> Thanks, >> >> Fande >> >> On Wed, Nov 10, 2021 at 12:16 PM Satish Balay wrote: >> >>> You are using petsc-3.15.1 - and likely the mumps build change between >>> then and current 3.16. >>> >>> Can you use latest PETSc release? >>> >>> If not - Suggest removing --download-mumps= >>> https://bitbucket.org/petsc/pkg-mumps.git >>> --download-mumps-commit=v5.4.1-p1 options [and PETSC_ARCH] and going back >>> to your old >>> build. >>> >>> If it fails [as before] - retry with: --with-make-np=1 >>> >>> Satish >>> >>> On Wed, 10 Nov 2021, Novak, April via petsc-users wrote: >>> >>> > Hi Barry, >>> > >>> > Thank you for your assistance - I?ve attached the latest >>> configure.log. I still encounter issues building, though some of the MUMPS >>> errors do seem to have been fixed with the --download-mumps-commit option. >>> Do you have a recommendation for addressing these other errors? >>> > >>> > Thanks, >>> > >>> > -April >>> > >>> > From: Fande Kong >>> > Date: Wednesday, November 10, 2021 at 9:44 AM >>> > To: Barry Smith >>> > Cc: PETSc users list , Novak, April < >>> anovak at anl.gov> >>> > Subject: Re: [petsc-users] Error running make on MUMPS >>> > Thanks, Barry, >>> > >>> > We will try this, and report back >>> > >>> > >>> > Fande >>> > >>> > On Tue, Nov 9, 2021 at 5:41 PM Barry Smith >> bsmith at petsc.dev>> wrote: >>> > >>> > This version of MUMPS has a bug in its build system; it does not >>> have all the dependencies on Fortran modules properly listed so Fortran >>> files can get compiled too early causing "random" failures during some >>> builds, especially on machines with lots of cores for compiling. >>> > >>> > I think you should be able to use --download-mumps= >>> https://bitbucket.org/petsc/pkg-mumps.git >>> --download-mumps-commit=v5.4.1-p1 to get a patched version. >>> > >>> > Barry >>> > >>> > >>> > >>> > On Nov 9, 2021, at 6:10 PM, Fande Kong >> fdkong.jd at gmail.com>> wrote: >>> > >>> > Hi All, >>> > >>> > We encountered a configuration error when running the PETSc >>> configuration on a HPC system. Went through the log file, but could not >>> find much. The log file was attached. >>> > >>> > Any thoughts? >>> > >>> > Thanks for your help, as always. >>> > >>> > Fande >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Thu Nov 11 14:59:27 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 11 Nov 2021 15:59:27 -0500 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> <61ccdb88-8636-cf84-69f5-eda0a1a6294b@mcs.anl.gov> Message-ID: On Thu, Nov 11, 2021 at 3:44 PM Fande Kong wrote: > Thanks Matt, > > I understand completely, the actual error should be > > " > ln -s libHYPRE_parcsr_ls-2.20.0.so libHYPRE_parcsr_ls.so gmake[1]: > Leaving directory > `/beegfs1/home/anovak/cardinal/contrib/moose/petsc/arch-moose/externalpackages/git.hypre/src/parcsr_ls' > Error running make; make install on HYPRE: Could not execute > "['/usr/bin/gmake install']": > " > > The one you saw was that hypre automatically took the second try for > parcsr_ls after the first failed gmake install. Because the first try > already did "ln -s libHYPRE_parcsr_ls-2.20.0.so libHYPRE_parcsr_ls.so", > and then second try would see that ?libHYPRE_parcsr_ls.so" already existed. > Are you completely sure? This just looks like a cascading make error, namely that one shell thing failed (ln) and then the make reports an error. Thanks, Matt > Thanks, > Fande > > On Thu, Nov 11, 2021 at 1:29 PM Matthew Knepley wrote: > >> On Thu, Nov 11, 2021 at 3:25 PM Fande Kong wrote: >> >>> Thanks, Satish >>> >>> "--with-make-np=1" did help us on MUMPS, but we had new trouble with >>> hypre now. >>> >>> It is hard to understand why "gmake install" even failed. >>> >> >> Because HYPRE thinks it is better to use 'ln' than the 'install' script >> that handles things like the target existing: >> >> gmake[1]: Leaving directory >> `/beegfs1/home/anovak/cardinal/contrib/moose/petsc/arch-moose/externalpackages/git.hypre/src/parcsr_ls' >> ln: failed to create symbolic link ?libHYPRE_parcsr_ls.so?: File exists >> gmake[1]: *** [libHYPRE_parcsr_ls.so] Error 1 >> gmake: *** [all] Error 1 >> >> I think everything needs to be cleaned out for Hypre to reinstall. >> >> Thanks, >> >> Matt >> >> >>> Please see the attachment for the log file. >>> >>> Thanks, >>> >>> Fande >>> >>> On Wed, Nov 10, 2021 at 12:16 PM Satish Balay wrote: >>> >>>> You are using petsc-3.15.1 - and likely the mumps build change between >>>> then and current 3.16. >>>> >>>> Can you use latest PETSc release? >>>> >>>> If not - Suggest removing --download-mumps= >>>> https://bitbucket.org/petsc/pkg-mumps.git >>>> --download-mumps-commit=v5.4.1-p1 options [and PETSC_ARCH] and going back >>>> to your old >>>> build. >>>> >>>> If it fails [as before] - retry with: --with-make-np=1 >>>> >>>> Satish >>>> >>>> On Wed, 10 Nov 2021, Novak, April via petsc-users wrote: >>>> >>>> > Hi Barry, >>>> > >>>> > Thank you for your assistance - I?ve attached the latest >>>> configure.log. I still encounter issues building, though some of the MUMPS >>>> errors do seem to have been fixed with the --download-mumps-commit option. >>>> Do you have a recommendation for addressing these other errors? >>>> > >>>> > Thanks, >>>> > >>>> > -April >>>> > >>>> > From: Fande Kong >>>> > Date: Wednesday, November 10, 2021 at 9:44 AM >>>> > To: Barry Smith >>>> > Cc: PETSc users list , Novak, April < >>>> anovak at anl.gov> >>>> > Subject: Re: [petsc-users] Error running make on MUMPS >>>> > Thanks, Barry, >>>> > >>>> > We will try this, and report back >>>> > >>>> > >>>> > Fande >>>> > >>>> > On Tue, Nov 9, 2021 at 5:41 PM Barry Smith >>> bsmith at petsc.dev>> wrote: >>>> > >>>> > This version of MUMPS has a bug in its build system; it does not >>>> have all the dependencies on Fortran modules properly listed so Fortran >>>> files can get compiled too early causing "random" failures during some >>>> builds, especially on machines with lots of cores for compiling. >>>> > >>>> > I think you should be able to use --download-mumps= >>>> https://bitbucket.org/petsc/pkg-mumps.git >>>> --download-mumps-commit=v5.4.1-p1 to get a patched version. >>>> > >>>> > Barry >>>> > >>>> > >>>> > >>>> > On Nov 9, 2021, at 6:10 PM, Fande Kong >>> fdkong.jd at gmail.com>> wrote: >>>> > >>>> > Hi All, >>>> > >>>> > We encountered a configuration error when running the PETSc >>>> configuration on a HPC system. Went through the log file, but could not >>>> find much. The log file was attached. >>>> > >>>> > Any thoughts? >>>> > >>>> > Thanks for your help, as always. >>>> > >>>> > Fande >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdkong.jd at gmail.com Thu Nov 11 15:15:27 2021 From: fdkong.jd at gmail.com (Fande Kong) Date: Thu, 11 Nov 2021 14:15:27 -0700 Subject: [petsc-users] Error running make on MUMPS In-Reply-To: References: <8EC285F2-2511-4103-BEB0-C46D17EBE9DA@petsc.dev> <61ccdb88-8636-cf84-69f5-eda0a1a6294b@mcs.anl.gov> Message-ID: On Thu, Nov 11, 2021 at 1:59 PM Matthew Knepley wrote: > On Thu, Nov 11, 2021 at 3:44 PM Fande Kong wrote: > >> Thanks Matt, >> >> I understand completely, the actual error should be >> >> " >> ln -s libHYPRE_parcsr_ls-2.20.0.so libHYPRE_parcsr_ls.so gmake[1]: >> Leaving directory >> `/beegfs1/home/anovak/cardinal/contrib/moose/petsc/arch-moose/externalpackages/git.hypre/src/parcsr_ls' >> Error running make; make install on HYPRE: Could not execute >> "['/usr/bin/gmake install']": >> " >> >> The one you saw was that hypre automatically took the second try for >> parcsr_ls after the first failed gmake install. Because the first try >> already did "ln -s libHYPRE_parcsr_ls-2.20.0.so libHYPRE_parcsr_ls.so", >> and then second try would see that ?libHYPRE_parcsr_ls.so" already existed. >> > > Are you completely sure? This just looks like a cascading make error, > namely that one shell thing failed (ln) and then the make reports an error. > Oh, sorry. It was a typo. I intended to say "If I understand correctly, the actual should be balabala .." Thanks for the explanation! I was confused by the double outputs. I got it now. Thanks, again Fande > > Thanks, > > Matt > > >> Thanks, >> Fande >> >> On Thu, Nov 11, 2021 at 1:29 PM Matthew Knepley >> wrote: >> >>> On Thu, Nov 11, 2021 at 3:25 PM Fande Kong wrote: >>> >>>> Thanks, Satish >>>> >>>> "--with-make-np=1" did help us on MUMPS, but we had new trouble with >>>> hypre now. >>>> >>>> It is hard to understand why "gmake install" even failed. >>>> >>> >>> Because HYPRE thinks it is better to use 'ln' than the 'install' script >>> that handles things like the target existing: >>> >>> gmake[1]: Leaving directory >>> `/beegfs1/home/anovak/cardinal/contrib/moose/petsc/arch-moose/externalpackages/git.hypre/src/parcsr_ls' >>> ln: failed to create symbolic link ?libHYPRE_parcsr_ls.so?: File exists >>> gmake[1]: *** [libHYPRE_parcsr_ls.so] Error 1 >>> gmake: *** [all] Error 1 >>> >>> I think everything needs to be cleaned out for Hypre to reinstall. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Please see the attachment for the log file. >>>> >>>> Thanks, >>>> >>>> Fande >>>> >>>> On Wed, Nov 10, 2021 at 12:16 PM Satish Balay >>>> wrote: >>>> >>>>> You are using petsc-3.15.1 - and likely the mumps build change between >>>>> then and current 3.16. >>>>> >>>>> Can you use latest PETSc release? >>>>> >>>>> If not - Suggest removing --download-mumps= >>>>> https://bitbucket.org/petsc/pkg-mumps.git >>>>> --download-mumps-commit=v5.4.1-p1 options [and PETSC_ARCH] and going back >>>>> to your old >>>>> build. >>>>> >>>>> If it fails [as before] - retry with: --with-make-np=1 >>>>> >>>>> Satish >>>>> >>>>> On Wed, 10 Nov 2021, Novak, April via petsc-users wrote: >>>>> >>>>> > Hi Barry, >>>>> > >>>>> > Thank you for your assistance - I?ve attached the latest >>>>> configure.log. I still encounter issues building, though some of the MUMPS >>>>> errors do seem to have been fixed with the --download-mumps-commit option. >>>>> Do you have a recommendation for addressing these other errors? >>>>> > >>>>> > Thanks, >>>>> > >>>>> > -April >>>>> > >>>>> > From: Fande Kong >>>>> > Date: Wednesday, November 10, 2021 at 9:44 AM >>>>> > To: Barry Smith >>>>> > Cc: PETSc users list , Novak, April < >>>>> anovak at anl.gov> >>>>> > Subject: Re: [petsc-users] Error running make on MUMPS >>>>> > Thanks, Barry, >>>>> > >>>>> > We will try this, and report back >>>>> > >>>>> > >>>>> > Fande >>>>> > >>>>> > On Tue, Nov 9, 2021 at 5:41 PM Barry Smith >>>> bsmith at petsc.dev>> wrote: >>>>> > >>>>> > This version of MUMPS has a bug in its build system; it does not >>>>> have all the dependencies on Fortran modules properly listed so Fortran >>>>> files can get compiled too early causing "random" failures during some >>>>> builds, especially on machines with lots of cores for compiling. >>>>> > >>>>> > I think you should be able to use --download-mumps= >>>>> https://bitbucket.org/petsc/pkg-mumps.git >>>>> --download-mumps-commit=v5.4.1-p1 to get a patched version. >>>>> > >>>>> > Barry >>>>> > >>>>> > >>>>> > >>>>> > On Nov 9, 2021, at 6:10 PM, Fande Kong >>>> fdkong.jd at gmail.com>> wrote: >>>>> > >>>>> > Hi All, >>>>> > >>>>> > We encountered a configuration error when running the PETSc >>>>> configuration on a HPC system. Went through the log file, but could not >>>>> find much. The log file was attached. >>>>> > >>>>> > Any thoughts? >>>>> > >>>>> > Thanks for your help, as always. >>>>> > >>>>> > Fande >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhargav.subramanya at kaust.edu.sa Thu Nov 11 16:24:50 2021 From: bhargav.subramanya at kaust.edu.sa (Bhargav Subramanya) Date: Fri, 12 Nov 2021 01:24:50 +0300 Subject: [petsc-users] dmplex extrude in parallel In-Reply-To: References: Message-ID: Dear Matt, Thanks for the reply. 1. I realized that, for some reason, all command line options don't seem to work for me. For the simple code that you mentioned (also shown below), the only command line options that I am successful are /home/subrambm/petsc/arch-linux-c-debug/bin/mpiexec -n 2 ./cavity_flow.out -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_view hdf5:mesh.h5 -dm_distribute 1 #include int main(int argc, char **argv) { DM dm; PetscErrorCode ierr; ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); ierr = DMSetFromOptions(dm);CHKERRQ(ierr); ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); ierr = DMDestroy(&dm);CHKERRQ(ierr); ierr = PetscFinalize(); return ierr; } 2. Command line options after distributing the mesh, like dm_refine gives error (without any extrusion code or commands) 3. Even with a single processor, although extrude does not give any errors, it gives a bad mesh data. I tried with .h5 file format as well. -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_view hdf5:mesh.h5 -dm_extrude_layers 4 -dm_extrude_thickness 0.1 -dm_extrude_column_first 0 ----------------------------------------------------------- Although this is not what I eventually want, only the following code with command line options (-dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_view hdf5:mesh.h5) is giving the prismatic mesh in parallel. ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); ierr = DMSetFromOptions(dm);CHKERRQ(ierr); { DM dm3D; ierr = DMPlexExtrude(dm, 4, 0.1, PETSC_FALSE, NULL, PETSC_TRUE, &dm3D); CHKERRQ(ierr); if (dm3D) { DMDestroy(&dm); dm = dm3D; } } { DM dmDist; ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); if (dmDist) { DMDestroy(&dm); dm = dmDist; } } ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); ----------------------------------------------------------- On Thu, Nov 11, 2021 at 9:41 PM Matthew Knepley wrote: > On Thu, Nov 11, 2021 at 1:17 PM Bhargav Subramanya < > bhargav.subramanya at kaust.edu.sa> wrote: > >> Dear Matt, >> >> I just realized that I used PETSC_COMM_SELF instead of PETSC_COMM_WORLD >> while performing the above test. I am sorry for that. After fixing it, I >> have the following 3 cases: >> >> 1. With the below command line options, where I simply create the >> spherical surface mesh, everything is fine. The mesh gets distributed in >> parallel with the check DMPlexIsDistributed. >> /home/subrambm/petsc/arch-linux-c-debug/bin/mpiexec -n 2 >> ./cavity_flow.out -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_shape sphere >> -dm_plex_sphere_radius 1.0 -dm_refine_pre 2 -dm_view vtk:mesh.vtk >> -dm_distribute 1 >> >> ------------------------------------------------------------------- >> 2. This case also seems to work. However, the mesh file data is bad. Do >> you think it's because of the vtk file format? >> -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 >> -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_extrude_layers 4 >> -dm_extrude_thickness 0.1 -dm_extrude_column_first 0 -dm_distribute 1 >> -dm_view vtk:mesh.vtk >> > > It should work, but that would be the first place I would check for a bug. > I would use > > -dm_view hdf5:mesh.h5 > > and then > > $PETSC_DIR/lib/petsc/bin/petsc_gen_xdmf.py mesh.h5 > > to make mesh.xmf, which can be loaded by ParaView. > -- this is very helpful. > >> ------------------------------------------------------------------- >> >> 3. This is the case which I want: To distribute the mesh, refine it in >> parallel and extrude it. >> However, after distribution, the mesh does not seem to be refined as I >> get the following error: >> -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 >> -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_distribute 1 -dm_refine 1 >> > > The problem here is that you are refining the extruded mesh, not refining > the surface mesh. There must be something switched around in your file. > --I was probably not clear here. I did not have anything related to extrusion in the code. I only had the command line options: -dm_plex_shape sphere -dm_plex_dim 2 -dm_plex_simplex 1 -dm_plex_sphere_radius 1.0 -dm_refine_pre 3 -dm_distribute 1 -dm_refine 1 As I mentioned above any refinement or extrusion after the mesh distribution fails for me. Thanks, > > Matt > > >> Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really >> needed). >> Ignoring PCI device with non-16bit domain. >> Pass --enable-32bits-pci-domain to configure to support such devices >> (warning: it would break the library ABI, don't enable unless really >> needed). >> [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [1]PETSC ERROR: Nonconforming object sizes >> [1]PETSC ERROR: The output section point (0) closure size 9 != dual space >> dimension 12 at height 0 in [0, 0] >> [1]PETSC ERROR: See https://petsc.org/release/faq/ >> >> for trouble shooting. >> [1]PETSC ERROR: Petsc Release Version 3.16.1, unknown >> [1]PETSC ERROR: ./cavity_flow.out on a arch-linux-c-debug named kw60970 >> by subrambm Thu Nov 11 21:13:27 2021 >> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack >> --download-superlu_dist --download-hypre --download-fiat >> --download-generator --download-triangle --download-tetgen --download-chaco >> --download-make -download-boost --download-cmake --download-ml >> --download-mumps --download-scalapack --download-parmetis --download-metis >> --download-ptscotch --download-hdf5 >> [1]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() at >> /home/subrambm/petsc/src/dm/impls/plex/plexproject.c:762 >> [1]PETSC ERROR: #2 DMProjectFieldLocal_Plex() at >> /home/subrambm/petsc/src/dm/impls/plex/plexproject.c:933 >> [1]PETSC ERROR: #3 DMProjectFieldLocal() at >> /home/subrambm/petsc/src/dm/interface/dm.c:8863 >> [1]PETSC ERROR: #4 DMPlexRemapGeometry() at >> /home/subrambm/petsc/src/dm/impls/plex/plexgeometry.c:3270 >> [1]PETSC ERROR: #5 DMSetFromOptions_Plex() at >> /home/subrambm/petsc/src/dm/impls/plex/plexcreate.c:3064 >> [1]PETSC ERROR: #6 DMSetFromOptions() at >> /home/subrambm/petsc/src/dm/interface/dm.c:902 >> [1]PETSC ERROR: #7 main() at meshtest.c:12 >> [1]PETSC ERROR: PETSc Option Table entries: >> [1]PETSC ERROR: -dm_distribute 1 >> [1]PETSC ERROR: -dm_plex_dim 2 >> [1]PETSC ERROR: -dm_plex_shape sphere >> [1]PETSC ERROR: -dm_plex_simplex 1 >> [1]PETSC ERROR: -dm_plex_sphere_radius 1.0 >> [1]PETSC ERROR: -dm_refine 1 >> [1]PETSC ERROR: -dm_refine_pre 3 >> [1]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov---------- >> application called MPI_Abort(MPI_COMM_WORLD, 60) - process 1 >> >> Thanks, >> Bhargav >> >> On Wed, Nov 10, 2021 at 4:33 PM Matthew Knepley >> wrote: >> >>> On Wed, Nov 10, 2021 at 8:26 AM Bhargav Subramanya < >>> bhargav.subramanya at kaust.edu.sa> wrote: >>> >>>> Dear Matt, >>>> >>>> Thanks a lot for the reply. I am now able to generate the prismatic >>>> mesh properly. >>>> >>> >>> Cool. >>> >>> >>>> In the case of mpiexec -n 2 ./meshtest -dm_plex_shape sphere >>>> -dm_refine_pre 3 -dm_distribute -dm_refine 2 and DMExtrude(), where I am >>>> ordering the extruded cells on the layers first; Is the mesh extruded in >>>> parallel for this case? >>>> >>> >>> Yes. >>> >>> >>>> The first time I distributed the mesh, it was only the spherical >>>> surface mesh. After dmplex extrude, the mesh changes. Do I need to >>>> redistribute the mesh again? >>>> >>> >>> Probably not. The balance factor will not change, meaning the relative >>> sizes of the partitions. However, if you have a lot of layers, maybe you >>> want to. It is easy, >>> just call DMDistribute() again. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> Thanks, >>>> Bhargav >>>> >>>> On Wed, Nov 10, 2021 at 12:56 AM Matthew Knepley >>>> wrote: >>>> >>>>> On Tue, Nov 9, 2021 at 9:54 AM Bhargav Subramanya < >>>>> bhargav.subramanya at kaust.edu.sa> wrote: >>>>> >>>>>> Dear All, >>>>>> >>>>>> I want to generate a prismatic mesh from a base spherical surface >>>>>> mesh. I first refine the base icosahedron sequential mesh to some extent, >>>>>> distribute the mesh, continue the refinement in parallel, project the mesh >>>>>> to a unit sphere, and then finally extrude the mesh. The following is the >>>>>> code I am using. As the extrusion is performed in parallel, the extruded >>>>>> mesh seems to be broken as shown in the attached figure. May I know how to >>>>>> get an intact extruded mesh in parallel? Also, is it possible to make mesh >>>>>> refinement respect the spherical surface geometry, without having to >>>>>> project it using a function as shown below? >>>>>> >>>>> >>>>> I think you can do most of what you want from the command line. Here >>>>> is a simple code: >>>>> >>>>> #include >>>>> >>>>> int main(int argc, char **argv) >>>>> { >>>>> DM dm; >>>>> PetscErrorCode ierr; >>>>> >>>>> ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return >>>>> ierr; >>>>> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); >>>>> ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); >>>>> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >>>>> ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); >>>>> ierr = DMDestroy(&dm);CHKERRQ(ierr); >>>>> ierr = PetscFinalize(); >>>>> return ierr; >>>>> } >>>>> >>>>> that I run using >>>>> >>>>> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre 3 >>>>> -dm_extrude 5 -dm_plex_transform_extrude_thickness 0.1 -dm_distribute >>>>> -dm_view hdf5:mesh.h5 >>>>> >>>>> and get the attached picture. I am refining before distribution only >>>>> here, since the after distribution refinement would refine the extrude >>>>> thing, which is not what you want. >>>>> If you want refinement after distribution, you could use this code to >>>>> do >>>>> >>>>> mpiexec -n 2 ./meshtest -dm_plex_shape sphere -dm_refine_pre >>>>> 3 -dm_distribute -dm_refine 2 >>>>> >>>>> and then call DMExtrude() by hand after that. Alternatively, I could >>>>> make a pre-extrusion refinement option. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> Thanks, >>>>>> Bhargav >>>>>> >>>>>> /* refine the sequential mesh first */ >>>>>> for (int r = 0; r < numRefinePre; ++r) { >>>>>> DM rdm = NULL; >>>>>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>>>>> DMDestroy(&dm); >>>>>> dm = rdm; >>>>>> } >>>>>> >>>>>> /* project to a unit sphere */ >>>>>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>>>>> >>>>>> /* create and distribute DM */ >>>>>> ierr = DMPlexDistribute(dm, 0, NULL, &dmDist);CHKERRQ(ierr); >>>>>> if (dmDist) { >>>>>> DMDestroy(&dm); >>>>>> dm = dmDist; >>>>>> } >>>>>> >>>>>> /* refine the mesh in parallel */ >>>>>> for (int r = 0; r < numRefine; ++r) { >>>>>> DM rdm = NULL; >>>>>> DMRefine(dm, MPI_COMM_WORLD, &rdm); >>>>>> DMDestroy(&dm); >>>>>> dm = rdm; >>>>>> } >>>>>> >>>>>> /* project to a unit sphere */ >>>>>> ierr = ProjectToUnitSphere(dm); CHKERRQ(ierr); >>>>>> >>>>>> ierr = DMPlexExtrude(dm, numLayers-1, radialThickness, PETSC_FALSE, >>>>>> NULL, PETSC_TRUE, &dm3D);CHKERRQ(ierr); >>>>>> >>>>>> if (dm3D) { >>>>>> DMDestroy(&dm); >>>>>> dm = dm3D; >>>>>> } >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> This message and its contents, including attachments are intended >>>>>> solely for the original recipient. If you are not the intended recipient or >>>>>> have received this message in error, please notify me immediately and >>>>>> delete this message from your computer system. Any unauthorized use or >>>>>> distribution is prohibited. Please consider the environment before printing >>>>>> this email. >>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> >>>>> >>>> >>>> ------------------------------ >>>> This message and its contents, including attachments are intended >>>> solely for the original recipient. If you are not the intended recipient or >>>> have received this message in error, please notify me immediately and >>>> delete this message from your computer system. Any unauthorized use or >>>> distribution is prohibited. Please consider the environment before printing >>>> this email. >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> ------------------------------ >> This message and its contents, including attachments are intended solely >> for the original recipient. If you are not the intended recipient or have >> received this message in error, please notify me immediately and delete >> this message from your computer system. Any unauthorized use or >> distribution is prohibited. Please consider the environment before printing >> this email. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From teddy.kord at gmail.com Sun Nov 14 10:15:14 2021 From: teddy.kord at gmail.com (Ted Kord) Date: Sun, 14 Nov 2021 16:15:14 +0000 Subject: [petsc-users] Unable to Install Kokkos and Find Cuda Compiler Message-ID: Dear all I am trying to install PETSc with Kokkos and with the CUDA option but I keep getting the following error: "UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Could not find a CUDA compiler. Please set with the option --with-cudac or -CUDAC and load the config.compilers module. " 'nvcc' is on my path and works fine. I have also tried specifying --with-cudac=/usr/local/cuda/bin/nvcc still with the same error. This is my 'configure line:' ./configure --download-openmpi --download-kokkos-kernels --download-kokkos-kernels-shared=1 --download-kokkos --with-cuda=1 --with-cudac=/usr/local/cuda/bin/nvcc --with-cuda-dir=/usr/local/cuda --download-viennacl --download-fblaslapack --download-adios-shared=1 --download-hdf5 --download-hdf5-shared-libraries=1 --download-adolc --download-colpack --download-hypre --download-hypre-shared=1 --download-codipack --download-metis --download-slepc --download-libceed --download-eigen --download-parmetis --download-ptscotch --download-mumps --download-scalapack --download-hwloc --download-pastix --download-random123 --download-suitesparse --download-superlu --download-superlu-shared=1 --download-superlu_dist --download-superlu_dist-shared=1 --download-trilinos --download-trilinos-shared=1 --download-ml --download-ml-shared=1 --download-fftw --download-fftw-shared=1 --download-gsl --download-magma --download-radau5 --download-sundials2 --download-tchem -- Best regards, Theodore -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Sun Nov 14 10:24:12 2021 From: bsmith at petsc.dev (Barry Smith) Date: Sun, 14 Nov 2021 11:24:12 -0500 Subject: [petsc-users] Unable to Install Kokkos and Find Cuda Compiler In-Reply-To: References: Message-ID: You need to send configure.log to petsc-maint at mcs.anl.gov so we can see why the PETSc configure CUDA checks failed. Once we have that we can either make configure more robust for the type of system you are working on or tell you what to change to get the configure to complete. If at all possible please use the main git branch of PETSc when running configure while working with GPUs. Barry > On Nov 14, 2021, at 11:15 AM, Ted Kord wrote: > > Dear all > > I am trying to install PETSc with Kokkos and with the CUDA option but I keep getting the following error: > > "UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Could not find a CUDA compiler. Please set with the option --with-cudac or -CUDAC and load the config.compilers module. > " > 'nvcc' is on my path and works fine. > > I have also tried specifying --with-cudac=/usr/local/cuda/bin/nvcc still with the same error. > > This is my 'configure line:' > > ./configure --download-openmpi --download-kokkos-kernels --download-kokkos-kernels-shared=1 --download-kokkos --with-cuda=1 --with-cudac=/usr/local/cuda/bin/nvcc --with-cuda-dir=/usr/local/cuda --download-viennacl --download-fblaslapack --download-adios-shared=1 --download-hdf5 --download-hdf5-shared-libraries=1 --download-adolc --download-colpack --download-hypre --download-hypre-shared=1 --download-codipack --download-metis --download-slepc --download-libceed --download-eigen --download-parmetis --download-ptscotch --download-mumps --download-scalapack --download-hwloc --download-pastix --download-random123 --download-suitesparse --download-superlu --download-superlu-shared=1 --download-superlu_dist --download-superlu_dist-shared=1 --download-trilinos --download-trilinos-shared=1 --download-ml --download-ml-shared=1 --download-fftw --download-fftw-shared=1 --download-gsl --download-magma --download-radau5 --download-sundials2 --download-tchem > > > -- > Best regards, > > Theodore > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Sun Nov 14 11:39:18 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Sun, 14 Nov 2021 17:39:18 +0000 Subject: [petsc-users] Creating rectangular submatrix Message-ID: Hello I need to extract a submatrix consisting of a number columns from a rectangular matrix. The documentation for MatCreateSubMatrix states that "each process should list the columns that will be in its "diagonal part" in the new matrix". However, is it not the case that not all columns are necessarily in some process's diagonal part? Consider the example of a 2x5 matrix stored on two processes. Each process then stores one row. Suppose we wish to create a submatrix with rows 0 and 1, and columns 2, 3, and 4 from the original matrix. The diagonal parts of the matrix created then each hold one element, namely (0,2) and (1,3), respectively (using source matrix indices). But how would the ISs for creating this submatrix look? Taking the syntax described in the documentation at face value tthe ISs would be row_is = [0 | 1] and col_is = [2 | 3]. Thus, there doesn't seem to be a way to tell MatCreateSubMatrix that we also want column 4 (unless we may simply let col_is = [2 | 3 4], and the new matrix' "diagonal part" becomes non-square - which might also explain the scare quotes around the term in the documentation?) I'd be surprised if the syntax does not allow certain submatrices to be formed; more likely I'm misunderstanding the documentation. Could someone shed some light on this? Thanks in advance. Best regards Peder [http://www.dtu.dk/-/media/DTU_Generelt/Andet/mail-signature-logo.png] Peder J?rgensgaard Olesen PhD student Department of Mechanical Engineering pjool at mek.dtu.dk Koppels All? Building 403, room 105 2800 Kgs. Lyngby www.dtu.dk/english -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Nov 14 12:17:07 2021 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 14 Nov 2021 13:17:07 -0500 Subject: [petsc-users] Creating rectangular submatrix In-Reply-To: References: Message-ID: On Sun, Nov 14, 2021 at 12:39 PM Peder J?rgensgaard Olesen via petsc-users < petsc-users at mcs.anl.gov> wrote: > Hello > > > I need to extract a submatrix consisting of a number columns from a > rectangular matrix. The documentation > for > MatCreateSubMatrix states that "each process should list the columns that > will be in its "diagonal part" in the new matrix". > > > However, is it not the case that not all columns are necessarily in some > process's diagonal part? Consider the example of a 2x5 matrix stored on > two processes. Each process then stores one row. Suppose we wish to create > a submatrix with rows 0 and 1, and columns 2, 3, and 4 from the original > matrix. The diagonal parts of the matrix created then each hold one > element, namely (0,2) and (1,3), respectively (using source matrix > indices). But how would the ISs for creating this submatrix look? > Taking the syntax described in the documentation at face value tthe ISs > would be row_is = [0 | 1] and col_is = [2 | 3]. Thus, there doesn't seem > to be a way to tell MatCreateSubMatrix that we also want column 4 (unless > we may simply let col_is = [2 | 3 4], and the new matrix' "diagonal part" > becomes non-square - which might also explain the scare quotes around the > term in the documentation?) > > > I'd be surprised if the syntax does not allow certain submatrices to be > formed; more likely I'm misunderstanding the documentation. Could someone > shed some light on this? > You are on the right track. You have to decide what processor is going to have each column. This is the partitioning of the left hand side vector will need when applying this new matrix. So [2 | 3 4] is valid. This would give you [01]* [03 04] [11] [13 14]* * diagonal part > > Thanks in advance. > > > Best regards > > Peder > > > > Peder J?rgensgaard Olesen > PhD student > Department of Mechanical Engineering > > pjool at mek.dtu.dk > Koppels All? > Building 403, room 105 > 2800 Kgs. Lyngby > www.dtu.dk/english > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjool at mek.dtu.dk Sun Nov 14 13:16:50 2021 From: pjool at mek.dtu.dk (=?iso-8859-1?Q?Peder_J=F8rgensgaard_Olesen?=) Date: Sun, 14 Nov 2021 19:16:50 +0000 Subject: [petsc-users] Creating rectangular submatrix In-Reply-To: References: , Message-ID: <78103943ab854db68133ff4ea9f770c4@mek.dtu.dk> I see, thanks. Is there some clever way to have PETSc decide for me which column goes where? ________________________________ Fra: Mark Adams Sendt: 14. november 2021 19:17:07 Til: Peder J?rgensgaard Olesen Cc: petsc-users at mcs.anl.gov Emne: Re: [petsc-users] Creating rectangular submatrix On Sun, Nov 14, 2021 at 12:39 PM Peder J?rgensgaard Olesen via petsc-users > wrote: Hello I need to extract a submatrix consisting of a number columns from a rectangular matrix. The documentation for MatCreateSubMatrix states that "each process should list the columns that will be in its "diagonal part" in the new matrix". However, is it not the case that not all columns are necessarily in some process's diagonal part? Consider the example of a 2x5 matrix stored on two processes. Each process then stores one row. Suppose we wish to create a submatrix with rows 0 and 1, and columns 2, 3, and 4 from the original matrix. The diagonal parts of the matrix created then each hold one element, namely (0,2) and (1,3), respectively (using source matrix indices). But how would the ISs for creating this submatrix look? Taking the syntax described in the documentation at face value tthe ISs would be row_is = [0 | 1] and col_is = [2 | 3]. Thus, there doesn't seem to be a way to tell MatCreateSubMatrix that we also want column 4 (unless we may simply let col_is = [2 | 3 4], and the new matrix' "diagonal part" becomes non-square - which might also explain the scare quotes around the term in the documentation?) I'd be surprised if the syntax does not allow certain submatrices to be formed; more likely I'm misunderstanding the documentation. Could someone shed some light on this? You are on the right track. You have to decide what processor is going to have each column. This is the partitioning of the left hand side vector will need when applying this new matrix. So [2 | 3 4] is valid. This would give you [01]* [03 04] [11] [13 14]* * diagonal part Thanks in advance. Best regards Peder [http://www.dtu.dk/-/media/DTU_Generelt/Andet/mail-signature-logo.png] Peder J?rgensgaard Olesen PhD student Department of Mechanical Engineering pjool at mek.dtu.dk Koppels All? Building 403, room 105 2800 Kgs. Lyngby www.dtu.dk/english -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Nov 14 13:26:49 2021 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 14 Nov 2021 14:26:49 -0500 Subject: [petsc-users] Creating rectangular submatrix In-Reply-To: <78103943ab854db68133ff4ea9f770c4@mek.dtu.dk> References: <78103943ab854db68133ff4ea9f770c4@mek.dtu.dk> Message-ID: On Sun, Nov 14, 2021 at 2:16 PM Peder J?rgensgaard Olesen wrote: > I see, thanks. > > > Is there some clever way to have PETSc decide for me which column goes > where? > You need to know what you want before you can find a clever way to get it. It is not clear what the issues are with your application, but the simplest thing is to just chop the result up evenly. You could make a vector of the number of columns and use PETSC_DECIDE for the number of local entries (n) in this new vector and give it the number of columns as the global size (N) and PETSc will just chop it. Then you could get the local range of the vector as your columns. But you could do that manually pretty easily. > ------------------------------ > *Fra:* Mark Adams > *Sendt:* 14. november 2021 19:17:07 > *Til:* Peder J?rgensgaard Olesen > *Cc:* petsc-users at mcs.anl.gov > *Emne:* Re: [petsc-users] Creating rectangular submatrix > > > > On Sun, Nov 14, 2021 at 12:39 PM Peder J?rgensgaard Olesen via petsc-users > wrote: > >> Hello >> >> >> I need to extract a submatrix consisting of a number columns from a >> rectangular matrix. The documentation >> for >> MatCreateSubMatrix states that "each process should list the columns that >> will be in its "diagonal part" in the new matrix". >> >> >> However, is it not the case that not all columns are necessarily in some >> process's diagonal part? Consider the example of a 2x5 matrix stored on >> two processes. Each process then stores one row. Suppose we wish to create >> a submatrix with rows 0 and 1, and columns 2, 3, and 4 from the original >> matrix. The diagonal parts of the matrix created then each hold one >> element, namely (0,2) and (1,3), respectively (using source matrix >> indices). But how would the ISs for creating this submatrix look? >> Taking the syntax described in the documentation at face value tthe ISs >> would be row_is = [0 | 1] and col_is = [2 | 3]. Thus, there doesn't seem >> to be a way to tell MatCreateSubMatrix that we also want column 4 >> (unless we may simply let col_is = [2 | 3 4], and the new matrix' >> "diagonal part" becomes non-square - which might also explain the >> scare quotes around the term in the documentation?) >> >> >> I'd be surprised if the syntax does not allow certain submatrices to be >> formed; more likely I'm misunderstanding the documentation. Could someone >> shed some light on this? >> > > You are on the right track. You have to decide what processor is going to > have each column. This is the partitioning of the left hand side vector > will need when applying this new matrix. > So [2 | 3 4] is valid. This would give you > [01]* [03 04] > [11] [13 14]* > * diagonal part > > >> >> Thanks in advance. >> >> >> Best regards >> >> Peder >> >> >> >> Peder J?rgensgaard Olesen >> PhD student >> Department of Mechanical Engineering >> >> pjool at mek.dtu.dk >> Koppels All? >> Building 403, room 105 >> 2800 Kgs. Lyngby >> www.dtu.dk/english >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From teddy.kord at gmail.com Sun Nov 14 13:19:00 2021 From: teddy.kord at gmail.com (Ted Kord) Date: Sun, 14 Nov 2021 19:19:00 +0000 Subject: [petsc-users] Unable to Install Kokkos and Find Cuda Compiler In-Reply-To: References: Message-ID: Attached is the configure.log. TK On Sun, 14 Nov 2021 at 16:24, Barry Smith wrote: > > You need to send configure.log to petsc-maint at mcs.anl.gov so we can see > why the PETSc configure CUDA checks failed. Once we have that we can either > make configure more robust for the type of system you are working on or > tell you what to change to get the configure to complete. If at all > possible please use the main git branch of PETSc when running configure > while working with GPUs. > > Barry > > > On Nov 14, 2021, at 11:15 AM, Ted Kord wrote: > > Dear all > > I am trying to install PETSc with Kokkos and with the CUDA option but I > keep getting the following error: > > "UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > Could not find a CUDA compiler. Please set with the option --with-cudac or > -CUDAC and load the config.compilers module. > " > 'nvcc' is on my path and works fine. > > I have also tried specifying --with-cudac=/usr/local/cuda/bin/nvcc still > with the same error. > > This is my 'configure line:' > > ./configure --download-openmpi --download-kokkos-kernels > --download-kokkos-kernels-shared=1 --download-kokkos --with-cuda=1 > --with-cudac=/usr/local/cuda/bin/nvcc --with-cuda-dir=/usr/local/cuda > --download-viennacl --download-fblaslapack --download-adios-shared=1 > --download-hdf5 --download-hdf5-shared-libraries=1 --download-adolc > --download-colpack --download-hypre --download-hypre-shared=1 > --download-codipack --download-metis --download-slepc --download-libceed > --download-eigen --download-parmetis --download-ptscotch --download-mumps > --download-scalapack --download-hwloc --download-pastix > --download-random123 --download-suitesparse --download-superlu > --download-superlu-shared=1 --download-superlu_dist > --download-superlu_dist-shared=1 --download-trilinos > --download-trilinos-shared=1 --download-ml --download-ml-shared=1 > --download-fftw --download-fftw-shared=1 --download-gsl --download-magma > --download-radau5 --download-sundials2 --download-tchem > > > -- > Best regards, > > Theodore > > > -- Best regards, Theodore -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: configure.log Type: text/x-log Size: 11597995 bytes Desc: not available URL: From mfadams at lbl.gov Sun Nov 14 17:52:17 2021 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 14 Nov 2021 18:52:17 -0500 Subject: [petsc-users] Unable to Install Kokkos and Find Cuda Compiler In-Reply-To: References: Message-ID: This seems to be the same issue on your other thread "Unable to Install Kokkos" On Sun, Nov 14, 2021 at 6:06 PM Ted Kord wrote: > Attached is the configure.log. > > TK > > On Sun, 14 Nov 2021 at 16:24, Barry Smith wrote: > >> >> You need to send configure.log to petsc-maint at mcs.anl.gov so we can >> see why the PETSc configure CUDA checks failed. Once we have that we can >> either make configure more robust for the type of system you are working on >> or tell you what to change to get the configure to complete. If at all >> possible please use the main git branch of PETSc when running configure >> while working with GPUs. >> >> Barry >> >> >> On Nov 14, 2021, at 11:15 AM, Ted Kord wrote: >> >> Dear all >> >> I am trying to install PETSc with Kokkos and with the CUDA option but I >> keep getting the following error: >> >> "UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >> details): >> >> ------------------------------------------------------------------------------- >> Could not find a CUDA compiler. Please set with the option --with-cudac >> or -CUDAC and load the config.compilers module. >> " >> 'nvcc' is on my path and works fine. >> >> I have also tried specifying --with-cudac=/usr/local/cuda/bin/nvcc still >> with the same error. >> >> This is my 'configure line:' >> >> ./configure --download-openmpi --download-kokkos-kernels >> --download-kokkos-kernels-shared=1 --download-kokkos --with-cuda=1 >> --with-cudac=/usr/local/cuda/bin/nvcc --with-cuda-dir=/usr/local/cuda >> --download-viennacl --download-fblaslapack --download-adios-shared=1 >> --download-hdf5 --download-hdf5-shared-libraries=1 --download-adolc >> --download-colpack --download-hypre --download-hypre-shared=1 >> --download-codipack --download-metis --download-slepc --download-libceed >> --download-eigen --download-parmetis --download-ptscotch --download-mumps >> --download-scalapack --download-hwloc --download-pastix >> --download-random123 --download-suitesparse --download-superlu >> --download-superlu-shared=1 --download-superlu_dist >> --download-superlu_dist-shared=1 --download-trilinos >> --download-trilinos-shared=1 --download-ml --download-ml-shared=1 >> --download-fftw --download-fftw-shared=1 --download-gsl --download-magma >> --download-radau5 --download-sundials2 --download-tchem >> >> >> -- >> Best regards, >> >> Theodore >> >> >> > > -- > Best regards, > > Theodore > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junchao.zhang at gmail.com Sun Nov 14 22:19:22 2021 From: junchao.zhang at gmail.com (Junchao Zhang) Date: Sun, 14 Nov 2021 22:19:22 -0600 Subject: [petsc-users] Unable to Install Kokkos and Find Cuda Compiler In-Reply-To: References: Message-ID: Hi Ted, I saw in configure.log, --with-cudac=/usr/local/cuda-11.4/bin/nvcc --with-cuda-dir=/usr/local/cuda I suppose /usr/local/cuda is just a symbol link to /usr/local/cuda-11.4/ I don't know why this error /usr/include/c++/11/cmath:45:15: fatal error: math.h: No such file or directory To know whether it was because petsc did not configure Kokkos correctly, or there was something wrong with your environment, you can download Kokkos and install yourself to see if you can succeed. Make sure /usr/local/cuda-11.4/bin/nvcc on your $PATH. For example, I download kokkos under my $HOME and install to $HOME/tmp/kokkos $ cd $HOME $ git clone https://github.com/kokkos/kokkos.git $ cd kokkos $ mkdir build $ cd build $ cmake ../ -DCMAKE_INSTALL_PREFIX=$HOME/tmp/kokkos -DCMAKE_CXX_COMPILER=g++ -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_CUDA=ON -DKokkos_ARCH_VOLTA75=ON -DKokkos_ENABLE_CUDA_LAMBDA=ON -DCMAKE_CXX_STANDARD=17 $ make -j install --Junchao Zhang On Sun, Nov 14, 2021 at 10:15 AM Ted Kord wrote: > Dear all > > I am trying to install PETSc with Kokkos and with the CUDA option but I > keep getting the following error: > > "UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > Could not find a CUDA compiler. Please set with the option --with-cudac or > -CUDAC and load the config.compilers module. > " > 'nvcc' is on my path and works fine. > > I have also tried specifying --with-cudac=/usr/local/cuda/bin/nvcc still > with the same error. > > This is my 'configure line:' > > ./configure --download-openmpi --download-kokkos-kernels > --download-kokkos-kernels-shared=1 --download-kokkos --with-cuda=1 > --with-cudac=/usr/local/cuda/bin/nvcc --with-cuda-dir=/usr/local/cuda > --download-viennacl --download-fblaslapack --download-adios-shared=1 > --download-hdf5 --download-hdf5-shared-libraries=1 --download-adolc > --download-colpack --download-hypre --download-hypre-shared=1 > --download-codipack --download-metis --download-slepc --download-libceed > --download-eigen --download-parmetis --download-ptscotch --download-mumps > --download-scalapack --download-hwloc --download-pastix > --download-random123 --download-suitesparse --download-superlu > --download-superlu-shared=1 --download-superlu_dist > --download-superlu_dist-shared=1 --download-trilinos > --download-trilinos-shared=1 --download-ml --download-ml-shared=1 > --download-fftw --download-fftw-shared=1 --download-gsl --download-magma > --download-radau5 --download-sundials2 --download-tchem > > > -- > Best regards, > > Theodore > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 15 02:55:32 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 15 Nov 2021 08:55:32 +0000 Subject: [petsc-users] Debugger won't start Message-ID: Hello, I am running PETSc 3.16.0 on a cluster compiled in debug mode using the flag -start_in_debugger. But I getting following error message: Unable to start debugger in xterm. No such file or directory. I think the program is searching in the wrong path, probably under /usr/bin but xterm is located in a different path. How do I set it to the right path? Best, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Mon Nov 15 03:04:08 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 15 Nov 2021 10:04:08 +0100 Subject: [petsc-users] Debugger won't start In-Reply-To: References: Message-ID: <7BFD3B81-424E-4D5C-820F-E18DAD3D1FCE@dsic.upv.es> If you are debugging a program with one MPI process, you can just do "-start_in_debugger noxterm". Otherwise, have a look at https://petsc.org/release/docs/manualpages/Sys/PetscSetDebugTerminal.html Jose > El 15 nov 2021, a las 9:55, Karthikeyan Chockalingam - STFC UKRI escribi?: > > Hello, > > I am running PETSc 3.16.0 on a cluster compiled in debug mode using the flag -start_in_debugger. But I getting following error message: > > Unable to start debugger in xterm. No such file or directory. > > I think the program is searching in the wrong path, probably under /usr/bin but xterm is located in a different path. > How do I set it to the right path? > > Best, > Karthik. > This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. > From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 15 04:06:36 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 15 Nov 2021 10:06:36 +0000 Subject: [petsc-users] Debugger won't start In-Reply-To: <7BFD3B81-424E-4D5C-820F-E18DAD3D1FCE@dsic.upv.es> References: <7BFD3B81-424E-4D5C-820F-E18DAD3D1FCE@dsic.upv.es> Message-ID: <50845740-5077-4233-B87B-88B19E233FC4@stfc.ac.uk> Thank you for your response. I am trying to debug in parallel and the used the flags in the link you sent me mpirun -np 2 ./ex45 -start_in_debugger -debug_terminal "screen -X -S debug screen" but the problem still persists. Now, I get a different message No screen session found. ?On 15/11/2021, 09:04, "Jose E. Roman" wrote: If you are debugging a program with one MPI process, you can just do "-start_in_debugger noxterm". Otherwise, have a look at https://petsc.org/release/docs/manualpages/Sys/PetscSetDebugTerminal.html Jose > El 15 nov 2021, a las 9:55, Karthikeyan Chockalingam - STFC UKRI escribi?: > > Hello, > > I am running PETSc 3.16.0 on a cluster compiled in debug mode using the flag -start_in_debugger. But I getting following error message: > > Unable to start debugger in xterm. No such file or directory. > > I think the program is searching in the wrong path, probably under /usr/bin but xterm is located in a different path. > How do I set it to the right path? > > Best, > Karthik. > This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. > From jroman at dsic.upv.es Mon Nov 15 04:50:35 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Mon, 15 Nov 2021 11:50:35 +0100 Subject: [petsc-users] Debugger won't start In-Reply-To: <50845740-5077-4233-B87B-88B19E233FC4@stfc.ac.uk> References: <7BFD3B81-424E-4D5C-820F-E18DAD3D1FCE@dsic.upv.es> <50845740-5077-4233-B87B-88B19E233FC4@stfc.ac.uk> Message-ID: <960487E2-2BB1-4CF4-B034-219CAAF46F4A@dsic.upv.es> -start_in_debugger -debug_terminal "/path/to/xterm -e" > El 15 nov 2021, a las 11:06, Karthikeyan Chockalingam - STFC UKRI escribi?: > > Thank you for your response. I am trying to debug in parallel and the used the flags in the link you sent me > > mpirun -np 2 ./ex45 -start_in_debugger -debug_terminal "screen -X -S debug screen" > > but the problem still persists. Now, I get a different message > > No screen session found. > > > ?On 15/11/2021, 09:04, "Jose E. Roman" wrote: > > If you are debugging a program with one MPI process, you can just do "-start_in_debugger noxterm". Otherwise, have a look at https://petsc.org/release/docs/manualpages/Sys/PetscSetDebugTerminal.html > > Jose > > >> El 15 nov 2021, a las 9:55, Karthikeyan Chockalingam - STFC UKRI escribi?: >> >> Hello, >> >> I am running PETSc 3.16.0 on a cluster compiled in debug mode using the flag -start_in_debugger. But I getting following error message: >> >> Unable to start debugger in xterm. No such file or directory. >> >> I think the program is searching in the wrong path, probably under /usr/bin but xterm is located in a different path. >> How do I set it to the right path? >> >> Best, >> Karthik. >> This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. >> > > From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 15 05:07:50 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 15 Nov 2021 11:07:50 +0000 Subject: [petsc-users] Debugger won't start In-Reply-To: <960487E2-2BB1-4CF4-B034-219CAAF46F4A@dsic.upv.es> References: <7BFD3B81-424E-4D5C-820F-E18DAD3D1FCE@dsic.upv.es> <50845740-5077-4233-B87B-88B19E233FC4@stfc.ac.uk> <960487E2-2BB1-4CF4-B034-219CAAF46F4A@dsic.upv.es> Message-ID: Thank you very much that worked! Best, Karthik. ?On 15/11/2021, 10:50, "Jose E. Roman" wrote: -start_in_debugger -debug_terminal "/path/to/xterm -e" > El 15 nov 2021, a las 11:06, Karthikeyan Chockalingam - STFC UKRI escribi?: > > Thank you for your response. I am trying to debug in parallel and the used the flags in the link you sent me > > mpirun -np 2 ./ex45 -start_in_debugger -debug_terminal "screen -X -S debug screen" > > but the problem still persists. Now, I get a different message > > No screen session found. > > > On 15/11/2021, 09:04, "Jose E. Roman" wrote: > > If you are debugging a program with one MPI process, you can just do "-start_in_debugger noxterm". Otherwise, have a look at https://petsc.org/release/docs/manualpages/Sys/PetscSetDebugTerminal.html > > Jose > > >> El 15 nov 2021, a las 9:55, Karthikeyan Chockalingam - STFC UKRI escribi?: >> >> Hello, >> >> I am running PETSc 3.16.0 on a cluster compiled in debug mode using the flag -start_in_debugger. But I getting following error message: >> >> Unable to start debugger in xterm. No such file or directory. >> >> I think the program is searching in the wrong path, probably under /usr/bin but xterm is located in a different path. >> How do I set it to the right path? >> >> Best, >> Karthik. >> This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. >> > > From teddy.kord at gmail.com Mon Nov 15 14:11:19 2021 From: teddy.kord at gmail.com (Ted Kord) Date: Mon, 15 Nov 2021 20:11:19 +0000 Subject: [petsc-users] Unable to Install Kokkos and Find Cuda Compiler In-Reply-To: References: Message-ID: Hi Junchao Thanks, I will try that. On Mon, 15 Nov 2021 at 04:19, Junchao Zhang wrote: > Hi Ted, > I saw in configure.log, > --with-cudac=/usr/local/cuda-11.4/bin/nvcc > --with-cuda-dir=/usr/local/cuda > I suppose /usr/local/cuda is just a symbol link to /usr/local/cuda-11.4/ > I don't know why this error > /usr/include/c++/11/cmath:45:15: fatal error: math.h: No such file > or directory > To know whether it was because petsc did not configure Kokkos correctly, > or there was something wrong with your environment, you can download > Kokkos and install yourself to see if you can succeed. Make sure > /usr/local/cuda-11.4/bin/nvcc on your $PATH. > > For example, I download kokkos under my $HOME and install to > $HOME/tmp/kokkos > > $ cd $HOME > $ git clone https://github.com/kokkos/kokkos.git > $ cd kokkos > $ mkdir build > $ cd build > $ cmake ../ -DCMAKE_INSTALL_PREFIX=$HOME/tmp/kokkos > -DCMAKE_CXX_COMPILER=g++ -DKokkos_ENABLE_SERIAL=ON -DKokkos_ENABLE_CUDA=ON > -DKokkos_ARCH_VOLTA75=ON -DKokkos_ENABLE_CUDA_LAMBDA=ON > -DCMAKE_CXX_STANDARD=17 > $ make -j install > > --Junchao Zhang > > > On Sun, Nov 14, 2021 at 10:15 AM Ted Kord wrote: > >> Dear all >> >> I am trying to install PETSc with Kokkos and with the CUDA option but I >> keep getting the following error: >> >> "UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for >> details): >> >> ------------------------------------------------------------------------------- >> Could not find a CUDA compiler. Please set with the option --with-cudac >> or -CUDAC and load the config.compilers module. >> " >> 'nvcc' is on my path and works fine. >> >> I have also tried specifying --with-cudac=/usr/local/cuda/bin/nvcc still >> with the same error. >> >> This is my 'configure line:' >> >> ./configure --download-openmpi --download-kokkos-kernels >> --download-kokkos-kernels-shared=1 --download-kokkos --with-cuda=1 >> --with-cudac=/usr/local/cuda/bin/nvcc --with-cuda-dir=/usr/local/cuda >> --download-viennacl --download-fblaslapack --download-adios-shared=1 >> --download-hdf5 --download-hdf5-shared-libraries=1 --download-adolc >> --download-colpack --download-hypre --download-hypre-shared=1 >> --download-codipack --download-metis --download-slepc --download-libceed >> --download-eigen --download-parmetis --download-ptscotch --download-mumps >> --download-scalapack --download-hwloc --download-pastix >> --download-random123 --download-suitesparse --download-superlu >> --download-superlu-shared=1 --download-superlu_dist >> --download-superlu_dist-shared=1 --download-trilinos >> --download-trilinos-shared=1 --download-ml --download-ml-shared=1 >> --download-fftw --download-fftw-shared=1 --download-gsl --download-magma >> --download-radau5 --download-sundials2 --download-tchem >> >> >> -- >> Best regards, >> >> Theodore >> >> -- Best regards, Theodore -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhfreewill at gmail.com Mon Nov 15 20:51:56 2021 From: zhfreewill at gmail.com (zhfreewill) Date: Tue, 16 Nov 2021 10:51:56 +0800 Subject: [petsc-users] How to apply boundary conditions when using TS? Message-ID: Hi, I'm new to PETSc. I have been confused about how to apply boundary conditions (BCs) (e.g., the Dirichlet and the Neumann type) when using the time-stepping (TS) object of PETSc. I have read the example codes ( https://petsc.org/release/src/ts/tutorials/index.html) and vaguely found that BCs seems to be set as follows: 1. when evaluating a matrix of the right hand side function g(x) (e.g., using a function like RHSMatrixHeat) 2. when evaluating a matrix of the Jacobian matrix of g'(x) (e.g., using a function like FormJacobian) For instance, in the ex1.4 ( https://petsc.org/release/src/ts/tutorials/ex4.c.html), with comments at the beginning states 17: /* ------------------------------------------------------------------------ 19: This program solves the one-dimensional heat equation (also called the 20: diffusion equation), 21: u_t = u_xx, 22: on the domain 0 <= x <= 1, with the boundary conditions 23: u(t,0) = 0, u(t,1) = 0, 24: and the initial condition 25: u(0,x) = sin(6*pi*x) + 3*sin(2*pi*x). 26: This is a linear, second-order, parabolic equation. 28: We discretize the right-hand side using finite differences with 29: uniform grid spacing h: 30: u_xx = (u_{i+1} - 2u_{i} + u_{i-1})/(h^2) ... 47: ------------------------------------------------------------------------- */ and Dirichlet BCs, u(t,0) = 0 and u(t,1) = 0, are to set on the both side of the discretized domain, the corresponding codes can be found on the following lines in a function RHSMatrixHeat: /* ------------------------------------------------------------------------- */ 463: RHSMatrixHeat - User-provided routine to compute the right-hand-side matrix for the heat equation. */ 491: PetscErrorCode RHSMatrixHeat(TS ts,PetscReal t,Vec X,Mat AA,Mat BB,void *ctx) 492: { ... 505: /* Set matrix rows corresponding to boundary data */ 509: if (mstart == 0) { /* first processor only */ 510: v[0] = 1.0; 511: MatSetValues(A,1,&mstart,1,&mstart,v,INSERT_VALUES); 512: mstart++; 513: } 515: if (mend == appctx->m) { /* last processor only */ 516: mend--; 517: v[0] = 1.0; 518: MatSetValues(A,1,&mend,1,&mend,v,INSERT_VALUES); 519: } 521: /* Set matrix rows corresponding to interior data. We construct the matrix one row at a time. */ 525: v[0] = sone; v[1] = stwo; v[2] = sone; 526: for (i=mstart; i From knepley at gmail.com Tue Nov 16 05:11:28 2021 From: knepley at gmail.com (Matthew Knepley) Date: Tue, 16 Nov 2021 06:11:28 -0500 Subject: [petsc-users] How to apply boundary conditions when using TS? In-Reply-To: References: Message-ID: On Mon, Nov 15, 2021 at 10:24 PM zhfreewill wrote: > Hi, > I'm new to PETSc. I have been confused about how to apply boundary > conditions (BCs) (e.g., the Dirichlet and the Neumann type) when using the > time-stepping (TS) object of PETSc. > > I have read the example codes ( > https://petsc.org/release/src/ts/tutorials/index.html) and vaguely found > that BCs seems to be set as follows: > > 1. when evaluating a matrix of the right hand side function g(x) (e.g., > using a function like RHSMatrixHeat) > 2. when evaluating a matrix of the Jacobian matrix of g'(x) (e.g., using a > function like FormJacobian) > > For instance, in the ex1.4 ( > https://petsc.org/release/src/ts/tutorials/ex4.c.html), with comments at > the beginning states > > 17: /* > ------------------------------------------------------------------------ > 19: This program solves the one-dimensional heat equation (also called > the > 20: diffusion equation), > 21: u_t = u_xx, > 22: on the domain 0 <= x <= 1, with the boundary conditions > 23: u(t,0) = 0, u(t,1) = 0, > 24: and the initial condition > 25: u(0,x) = sin(6*pi*x) + 3*sin(2*pi*x). > 26: This is a linear, second-order, parabolic equation. > > 28: We discretize the right-hand side using finite differences with > 29: uniform grid spacing h: > 30: u_xx = (u_{i+1} - 2u_{i} + u_{i-1})/(h^2) > ... > 47: > ------------------------------------------------------------------------- */ > > and Dirichlet BCs, u(t,0) = 0 and u(t,1) = 0, are to set on the both side > of the discretized domain, the corresponding codes can be found on the > following lines in a function RHSMatrixHeat: > > /* > ------------------------------------------------------------------------- */ > 463: RHSMatrixHeat - User-provided routine to compute the right-hand-side > matrix for the heat equation. */ > 491: PetscErrorCode RHSMatrixHeat(TS ts,PetscReal t,Vec X,Mat AA,Mat > BB,void *ctx) > 492: { > ... > 505: /* Set matrix rows corresponding to boundary data */ > 509: if (mstart == 0) { /* first processor only */ > 510: v[0] = 1.0; > 511: MatSetValues(A,1,&mstart,1,&mstart,v,INSERT_VALUES); > 512: mstart++; > 513: } > 515: if (mend == appctx->m) { /* last processor only */ > 516: mend--; > 517: v[0] = 1.0; > 518: MatSetValues(A,1,&mend,1,&mend,v,INSERT_VALUES); > 519: } > > 521: /* Set matrix rows corresponding to interior data. We construct > the matrix one row at a time. */ > 525: v[0] = sone; v[1] = stwo; v[2] = sone; > 526: for (i=mstart; i 527: idx[0] = i-1; idx[1] = i; idx[2] = i+1; > 528: MatSetValues(A,1,&i,3,idx,v,INSERT_VALUES); > 529: } > ... > 550: } > /* > ------------------------------------------------------------------------- */ > > > My questions are: > > 1. How do these lines of code embody the Dirichlet BCs u(t,0) = 0 and > u(t,1) = 0. > This is the finite difference version of BC. It is just moving the term for a known value to the forcing. However, since the BC is 0, then we just do not compute the term. If the BC were nonzero, you would also see the term in the residual. For example, https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex19.c#L271 > 2. Similarly, if I want to apply a Neumann type BCs in these lines, How > can I do? > For finite differences, you usually just approximate the derivative as the one-sided difference of the last two values, and set this equal to the known value. For finite elements, Neumann conditions are just an addition weak form integrated over the boundary. Thanks, Matt > Any suggestions or documents? > > Best, > Qw > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.levrero-florencio at onscale.com Tue Nov 16 10:19:57 2021 From: f.levrero-florencio at onscale.com (Francesc Levrero-Florencio) Date: Tue, 16 Nov 2021 16:19:57 +0000 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS Message-ID: Dear PETSc team and users, We are running a simple cantilever beam bending, where the profile of the beam is I-shaped, where we apply a bending force on one end and fully constrained displacements on the other end. The formulation is a large strain formulation in Total Lagrangian form, where the material of the beam is a Saint Venant-Kirchhoff hyperelastic material that uses the same constants as steel (200E9 GPa Young?s modulus and 0.3 Poisson?s ratio). The simulation finishes in the requested number of time steps by using the ?basic? type of line-search in the SNES (i.e. traditional Newton method without line-search) in a reasonable number of Newton iterations per time step (3 or 4 iterations). However, when using the ?bt? (or ?l2?, and in fact no type of line-search ends up yielding convergence) line-search type, the convergence never happens within the SNES default maximum number of iterations of 50. During solving with traditional Newton, the general behaviour of each time step is that the norm of the residual increases on the second call to the residual function, but then hugely decreases in the following one or two, up to the point where convergence is achieved. Using ?bt? line-search, the line-search discards the step at lambda=1 immediately because the norm of the residual is higher than that produced in the first call to the residual function, cutting down the value of lambda to a value significantly lower than 1. The simulation then progresses in following Newton iterations in a similar fashion, the line-search step at lambda=1 is always discarded, and then smaller steps are taken but convergence never occurs, for even the first time step. Here are a few values of the relevant norms (using traditional Newton) in the first time step: BASIC NEWTON LS Norm of the internal forces is 0 Norm of the external forces is 1374.49 Norm of the residual is 1374.49 Norm of the solution with Dirichlet BCs is 0 Number of SNES iteration is 0 --------------------------------------------------------------------- Norm of the internal forces is 113498 Norm of the external forces is 1374.49 Norm of the residual is 105053 Norm of the solution with Dirichlet BCs is 0.441466 Number of SNES iteration is 0 --------------------------------------------------------------------- Norm of the internal forces is 42953.5 Norm of the external forces is 1374.49 Norm of the residual is 11.3734 Norm of the solution with Dirichlet BCs is 0.441438 Number of SNES iteration is 1 Here are a few values of the relevant norms (using ?bt? line-search) in the first time step: BT NEWTONLS Norm of the internal forces is 0 Norm of the external forces is 1374.49 Norm of the residual is 1374.49 Norm of the solution with Dirichlet BCs is 0 Number of SNES iteration is 0 --------------------------------------------------------------------- Norm of the internal forces is 113498 Norm of the external forces is 1374.49 Norm of the residual is 105053 Norm of the solution with Dirichlet BCs is 0.441466 Number of SNES iteration is 0 (I assume this is the first try at lambda=1) --------------------------------------------------------------------- Norm of the internal forces is 4422.12 Norm of the external forces is 1374.49 Norm of the residual is 1622.74 Norm of the solution with Dirichlet BCs is 0.0441466 Number of SNES iteration is 0 Line search: gnorm after quadratic fit 1.622742343614e+03 (I assume that in this line-search iteration 0.05 < lambda < 1, but the corresponding residual is not smaller than the one in the first call, 1622 > 1374) --------------------------------------------------------------------- Norm of the internal forces is 2163.76 Norm of the external forces is 1374.49 Norm of the residual is 1331.88 Norm of the solution with Dirichlet BCs is 0.0220733 Number of SNES iteration is 0 Line search: Cubically determined step, current gnorm 1.331884625811e+03 lambda=5.0000000000000003e-02 (This is the accepted lambda for the current Newton iteration) --------------------------------------------------------------------- Norm of the internal forces is 104020 Norm of the external forces is 1374.49 Norm of the residual is 94739 Norm of the solution with Dirichlet BCs is 0.441323 Number of SNES iteration is 1 My question would be, any idea on how to deal with this situation? I would imagine a ?hack? would be to bypass the first residual norm, and have the line-search use the following one as the ?base norm? to check its reduction in further iterations, but we are open to any ideas. Thanks for your help in advance and please keep up the good work! Regards, Francesc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Nov 16 11:38:33 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 16 Nov 2021 12:38:33 -0500 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS In-Reply-To: References: Message-ID: Perhaps this behavior is the result of a "scaling" issue in how various terms affect the residual? In particular perhaps the terms for enforcing boundary conditions are scaled differently than terms for the PDE enforcement? > On Nov 16, 2021, at 11:19 AM, Francesc Levrero-Florencio wrote: > > Dear PETSc team and users, > > We are running a simple cantilever beam bending, where the profile of the beam is I-shaped, where we apply a bending force on one end and fully constrained displacements on the other end. The formulation is a large strain formulation in Total Lagrangian form, where the material of the beam is a Saint Venant-Kirchhoff hyperelastic material that uses the same constants as steel (200E9 GPa Young?s modulus and 0.3 Poisson?s ratio). > > The simulation finishes in the requested number of time steps by using the ?basic? type of line-search in the SNES (i.e. traditional Newton method without line-search) in a reasonable number of Newton iterations per time step (3 or 4 iterations). However, when using the ?bt? (or ?l2?, and in fact no type of line-search ends up yielding convergence) line-search type, the convergence never happens within the SNES default maximum number of iterations of 50. > > During solving with traditional Newton, the general behaviour of each time step is that the norm of the residual increases on the second call to the residual function, but then hugely decreases in the following one or two, up to the point where convergence is achieved. Using ?bt? line-search, the line-search discards the step at lambda=1 immediately because the norm of the residual is higher than that produced in the first call to the residual function, cutting down the value of lambda to a value significantly lower than 1. The simulation then progresses in following Newton iterations in a similar fashion, the line-search step at lambda=1 is always discarded, and then smaller steps are taken but convergence never occurs, for even the first time step. > > Here are a few values of the relevant norms (using traditional Newton) in the first time step: > > BASIC NEWTON LS > Norm of the internal forces is 0 > Norm of the external forces is 1374.49 > Norm of the residual is 1374.49 > Norm of the solution with Dirichlet BCs is 0 > Number of SNES iteration is 0 > --------------------------------------------------------------------- > Norm of the internal forces is 113498 > Norm of the external forces is 1374.49 > Norm of the residual is 105053 > Norm of the solution with Dirichlet BCs is 0.441466 > Number of SNES iteration is 0 > --------------------------------------------------------------------- > Norm of the internal forces is 42953.5 > Norm of the external forces is 1374.49 > Norm of the residual is 11.3734 > Norm of the solution with Dirichlet BCs is 0.441438 > Number of SNES iteration is 1 > > Here are a few values of the relevant norms (using ?bt? line-search) in the first time step: > > BT NEWTONLS > Norm of the internal forces is 0 > Norm of the external forces is 1374.49 > Norm of the residual is 1374.49 > Norm of the solution with Dirichlet BCs is 0 > Number of SNES iteration is 0 > --------------------------------------------------------------------- > Norm of the internal forces is 113498 > Norm of the external forces is 1374.49 > Norm of the residual is 105053 > Norm of the solution with Dirichlet BCs is 0.441466 > Number of SNES iteration is 0 > (I assume this is the first try at lambda=1) > --------------------------------------------------------------------- > Norm of the internal forces is 4422.12 > Norm of the external forces is 1374.49 > Norm of the residual is 1622.74 > Norm of the solution with Dirichlet BCs is 0.0441466 > Number of SNES iteration is 0 > Line search: gnorm after quadratic fit 1.622742343614e+03 > (I assume that in this line-search iteration 0.05 < lambda < 1, but the corresponding residual is not smaller than the one in the first call, 1622 > 1374) > --------------------------------------------------------------------- > Norm of the internal forces is 2163.76 > Norm of the external forces is 1374.49 > Norm of the residual is 1331.88 > Norm of the solution with Dirichlet BCs is 0.0220733 > Number of SNES iteration is 0 > Line search: Cubically determined step, current gnorm 1.331884625811e+03 lambda=5.0000000000000003e-02 > (This is the accepted lambda for the current Newton iteration) > --------------------------------------------------------------------- > Norm of the internal forces is 104020 > Norm of the external forces is 1374.49 > Norm of the residual is 94739 > Norm of the solution with Dirichlet BCs is 0.441323 > Number of SNES iteration is 1 > > My question would be, any idea on how to deal with this situation? I would imagine a ?hack? would be to bypass the first residual norm, and have the line-search use the following one as the ?base norm? to check its reduction in further iterations, but we are open to any ideas. > > Thanks for your help in advance and please keep up the good work! > > Regards, > Francesc. From f.levrero-florencio at onscale.com Tue Nov 16 12:37:32 2021 From: f.levrero-florencio at onscale.com (Francesc Levrero-Florencio) Date: Tue, 16 Nov 2021 18:37:32 +0000 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS In-Reply-To: References: Message-ID: Hi Barry, Thanks for the quick reply. I am not sure what you meant by "scaling". In our code, every time our residual is called we do the following in this order: - Apply updated Dirichlet BCs in order to assemble the internal forces, and add them to the residual. - Assemble the external forces, and subtract them from the residual. - Update the residual to take into account Dirichet BCs, so we set the corresponding indices of the residual to - (dirichlet_bc_value - solution_value), the latter obtained directly from the solution vector of PETSc. However, note that in this case all Dirichlet BCs are homogeneous, so this value would be just "solution_value", or 0 throughout the simulation. This ensures that the solution vector coming from PETSc has the Dirichlet BCs applied. This is basically what we do in the residual. If I am not mistaken, the lambda scaling factor from the line-search would scale the factor "jacobian^{-1} * residual", so it would scale equally both internal and external forces. Regards, Francesc. On Tue, Nov 16, 2021 at 5:38 PM Barry Smith wrote: > > Perhaps this behavior is the result of a "scaling" issue in how various > terms affect the residual? In particular perhaps the terms for enforcing > boundary conditions are scaled differently than terms for the PDE > enforcement? > > > > > On Nov 16, 2021, at 11:19 AM, Francesc Levrero-Florencio < > f.levrero-florencio at onscale.com> wrote: > > > > Dear PETSc team and users, > > > > We are running a simple cantilever beam bending, where the profile of > the beam is I-shaped, where we apply a bending force on one end and fully > constrained displacements on the other end. The formulation is a large > strain formulation in Total Lagrangian form, where the material of the beam > is a Saint Venant-Kirchhoff hyperelastic material that uses the same > constants as steel (200E9 GPa Young?s modulus and 0.3 Poisson?s ratio). > > > > The simulation finishes in the requested number of time steps by using > the ?basic? type of line-search in the SNES (i.e. traditional Newton method > without line-search) in a reasonable number of Newton iterations per time > step (3 or 4 iterations). However, when using the ?bt? (or ?l2?, and in > fact no type of line-search ends up yielding convergence) line-search type, > the convergence never happens within the SNES default maximum number of > iterations of 50. > > > > During solving with traditional Newton, the general behaviour of each > time step is that the norm of the residual increases on the second call to > the residual function, but then hugely decreases in the following one or > two, up to the point where convergence is achieved. Using ?bt? line-search, > the line-search discards the step at lambda=1 immediately because the norm > of the residual is higher than that produced in the first call to the > residual function, cutting down the value of lambda to a value > significantly lower than 1. The simulation then progresses in following > Newton iterations in a similar fashion, the line-search step at lambda=1 is > always discarded, and then smaller steps are taken but convergence never > occurs, for even the first time step. > > > > Here are a few values of the relevant norms (using traditional Newton) > in the first time step: > > > > BASIC NEWTON LS > > Norm of the internal forces is 0 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1374.49 > > Norm of the solution with Dirichlet BCs is 0 > > Number of SNES iteration is 0 > > --------------------------------------------------------------------- > > Norm of the internal forces is 113498 > > Norm of the external forces is 1374.49 > > Norm of the residual is 105053 > > Norm of the solution with Dirichlet BCs is 0.441466 > > Number of SNES iteration is 0 > > --------------------------------------------------------------------- > > Norm of the internal forces is 42953.5 > > Norm of the external forces is 1374.49 > > Norm of the residual is 11.3734 > > Norm of the solution with Dirichlet BCs is 0.441438 > > Number of SNES iteration is 1 > > > > Here are a few values of the relevant norms (using ?bt? line-search) in > the first time step: > > > > BT NEWTONLS > > Norm of the internal forces is 0 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1374.49 > > Norm of the solution with Dirichlet BCs is 0 > > Number of SNES iteration is 0 > > --------------------------------------------------------------------- > > Norm of the internal forces is 113498 > > Norm of the external forces is 1374.49 > > Norm of the residual is 105053 > > Norm of the solution with Dirichlet BCs is 0.441466 > > Number of SNES iteration is 0 > > (I assume this is the first try at lambda=1) > > --------------------------------------------------------------------- > > Norm of the internal forces is 4422.12 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1622.74 > > Norm of the solution with Dirichlet BCs is 0.0441466 > > Number of SNES iteration is 0 > > Line search: gnorm after quadratic fit 1.622742343614e+03 > > (I assume that in this line-search iteration 0.05 < lambda < 1, but the > corresponding residual is not smaller than the one in the first call, 1622 > > 1374) > > --------------------------------------------------------------------- > > Norm of the internal forces is 2163.76 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1331.88 > > Norm of the solution with Dirichlet BCs is 0.0220733 > > Number of SNES iteration is 0 > > Line search: Cubically determined step, current gnorm 1.331884625811e+03 > lambda=5.0000000000000003e-02 > > (This is the accepted lambda for the current Newton iteration) > > --------------------------------------------------------------------- > > Norm of the internal forces is 104020 > > Norm of the external forces is 1374.49 > > Norm of the residual is 94739 > > Norm of the solution with Dirichlet BCs is 0.441323 > > Number of SNES iteration is 1 > > > > My question would be, any idea on how to deal with this situation? I > would imagine a ?hack? would be to bypass the first residual norm, and have > the line-search use the following one as the ?base norm? to check its > reduction in further iterations, but we are open to any ideas. > > > > Thanks for your help in advance and please keep up the good work! > > > > Regards, > > Francesc. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Tue Nov 16 12:54:59 2021 From: bsmith at petsc.dev (Barry Smith) Date: Tue, 16 Nov 2021 13:54:59 -0500 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS In-Reply-To: References: Message-ID: <89B70D15-8949-45BF-B0D3-3B605AF84EA0@petsc.dev> By "scaling" I mean you may need to rescale the residual terms from the Dirichlet boundary conditions to be the same order (in terms of the characteristic size of the elements) as they are for the other residuals. It seems in your formulation the Jacobian entries of the residual for Dirichlet boundary conditions are 1 on the diagonal while the Jacobian entries of the residual for the PDE entries will have a scaling related to the finite element method you are using, for example in 3d with an operator grad dot grad the Jacobian entry will be order dx = (dx)^3 * (1/dx)^2. Thus you should scale the Dirichlet boundary condition residuals by dx to get the same scaling. Barry > On Nov 16, 2021, at 1:37 PM, Francesc Levrero-Florencio wrote: > > Hi Barry, > > Thanks for the quick reply. I am not sure what you meant by "scaling". In our code, every time our residual is called we do the following in this order: > - Apply updated Dirichlet BCs in order to assemble the internal forces, and add them to the residual. > - Assemble the external forces, and subtract them from the residual. > - Update the residual to take into account Dirichet BCs, so we set the corresponding indices of the residual to - (dirichlet_bc_value - solution_value), the latter obtained directly from the solution vector of PETSc. However, note that in this case all Dirichlet BCs are homogeneous, so this value would be just "solution_value", or 0 throughout the simulation. This ensures that the solution vector coming from PETSc has the Dirichlet BCs applied. > > This is basically what we do in the residual. If I am not mistaken, the lambda scaling factor from the line-search would scale the factor "jacobian^{-1} * residual", so it would scale equally both internal and external forces. > > Regards, > Francesc. > > On Tue, Nov 16, 2021 at 5:38 PM Barry Smith > wrote: > > Perhaps this behavior is the result of a "scaling" issue in how various terms affect the residual? In particular perhaps the terms for enforcing boundary conditions are scaled differently than terms for the PDE enforcement? > > > > > On Nov 16, 2021, at 11:19 AM, Francesc Levrero-Florencio > wrote: > > > > Dear PETSc team and users, > > > > We are running a simple cantilever beam bending, where the profile of the beam is I-shaped, where we apply a bending force on one end and fully constrained displacements on the other end. The formulation is a large strain formulation in Total Lagrangian form, where the material of the beam is a Saint Venant-Kirchhoff hyperelastic material that uses the same constants as steel (200E9 GPa Young?s modulus and 0.3 Poisson?s ratio). > > > > The simulation finishes in the requested number of time steps by using the ?basic? type of line-search in the SNES (i.e. traditional Newton method without line-search) in a reasonable number of Newton iterations per time step (3 or 4 iterations). However, when using the ?bt? (or ?l2?, and in fact no type of line-search ends up yielding convergence) line-search type, the convergence never happens within the SNES default maximum number of iterations of 50. > > > > During solving with traditional Newton, the general behaviour of each time step is that the norm of the residual increases on the second call to the residual function, but then hugely decreases in the following one or two, up to the point where convergence is achieved. Using ?bt? line-search, the line-search discards the step at lambda=1 immediately because the norm of the residual is higher than that produced in the first call to the residual function, cutting down the value of lambda to a value significantly lower than 1. The simulation then progresses in following Newton iterations in a similar fashion, the line-search step at lambda=1 is always discarded, and then smaller steps are taken but convergence never occurs, for even the first time step. > > > > Here are a few values of the relevant norms (using traditional Newton) in the first time step: > > > > BASIC NEWTON LS > > Norm of the internal forces is 0 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1374.49 > > Norm of the solution with Dirichlet BCs is 0 > > Number of SNES iteration is 0 > > --------------------------------------------------------------------- > > Norm of the internal forces is 113498 > > Norm of the external forces is 1374.49 > > Norm of the residual is 105053 > > Norm of the solution with Dirichlet BCs is 0.441466 > > Number of SNES iteration is 0 > > --------------------------------------------------------------------- > > Norm of the internal forces is 42953.5 > > Norm of the external forces is 1374.49 > > Norm of the residual is 11.3734 > > Norm of the solution with Dirichlet BCs is 0.441438 > > Number of SNES iteration is 1 > > > > Here are a few values of the relevant norms (using ?bt? line-search) in the first time step: > > > > BT NEWTONLS > > Norm of the internal forces is 0 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1374.49 > > Norm of the solution with Dirichlet BCs is 0 > > Number of SNES iteration is 0 > > --------------------------------------------------------------------- > > Norm of the internal forces is 113498 > > Norm of the external forces is 1374.49 > > Norm of the residual is 105053 > > Norm of the solution with Dirichlet BCs is 0.441466 > > Number of SNES iteration is 0 > > (I assume this is the first try at lambda=1) > > --------------------------------------------------------------------- > > Norm of the internal forces is 4422.12 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1622.74 > > Norm of the solution with Dirichlet BCs is 0.0441466 > > Number of SNES iteration is 0 > > Line search: gnorm after quadratic fit 1.622742343614e+03 > > (I assume that in this line-search iteration 0.05 < lambda < 1, but the corresponding residual is not smaller than the one in the first call, 1622 > 1374) > > --------------------------------------------------------------------- > > Norm of the internal forces is 2163.76 > > Norm of the external forces is 1374.49 > > Norm of the residual is 1331.88 > > Norm of the solution with Dirichlet BCs is 0.0220733 > > Number of SNES iteration is 0 > > Line search: Cubically determined step, current gnorm 1.331884625811e+03 lambda=5.0000000000000003e-02 > > (This is the accepted lambda for the current Newton iteration) > > --------------------------------------------------------------------- > > Norm of the internal forces is 104020 > > Norm of the external forces is 1374.49 > > Norm of the residual is 94739 > > Norm of the solution with Dirichlet BCs is 0.441323 > > Number of SNES iteration is 1 > > > > My question would be, any idea on how to deal with this situation? I would imagine a ?hack? would be to bypass the first residual norm, and have the line-search use the following one as the ?base norm? to check its reduction in further iterations, but we are open to any ideas. > > > > Thanks for your help in advance and please keep up the good work! > > > > Regards, > > Francesc. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From berend.vanwachem at ovgu.de Wed Nov 17 09:16:46 2021 From: berend.vanwachem at ovgu.de (Berend van Wachem) Date: Wed, 17 Nov 2021 16:16:46 +0100 Subject: [petsc-users] DMView and DMLoad In-Reply-To: <056E066F-D596-4254-A44A-60BFFD30FE82@erdw.ethz.ch> References: <56ce2135-9757-4292-e33b-c7eea8fb7b2e@ovgu.de> <056E066F-D596-4254-A44A-60BFFD30FE82@erdw.ethz.ch> Message-ID: <6c4e0656-db99-e9da-000f-ab9f7dd62c07@ovgu.de> Dear Vaclav, Lawrence, Koki, Thanks for your help! Following your advice and following your example (https://petsc.org/main/docs/manual/dmplex/#saving-and-loading-data-with-hdf5) we are able to save and load the DM with a wrapped Vector in h5 format (PETSC_VIEWER_HDF5_PETSC) successfully. For saving, we use something similar to: DMPlexTopologyView(dm, viewer); DMClone(dm, &sdm); ... DMPlexSectionView(dm, viewer, sdm); DMGetLocalVector(sdm, &vec); ... DMPlexLocalVectorView(dm, viewer, sdm, vec); and for loading: DMCreate(PETSC_COMM_WORLD, &dm); DMSetType(dm, DMPLEX); ... PetscViewerPushFormat(viewer, PETSC_VIEWER_HDF5_PETSC); DMPlexTopologyLoad(dm, viewer, &sfO); DMPlexLabelsLoad(dm, viewer); DMPlexCoordinatesLoad(dm, viewer); PetscViewerPopFormat(viewer); ... PetscSFCompose(sfO, sfDist, &sf); ... DMClone(dm, &sdm); DMPlexSectionLoad(dm, viewer, sdm, sf, &globalDataSF, &localDataSF); DMGetLocalVector(sdm, &vec); ... DMPlexLocalVectorLoad(dm, viewer, sdm, localDataSF, vec); This works fine for non-periodic DMs but for periodic cases the line: DMPlexCoordinatesLoad(dm, H5Viewer); delivers the error message: invalid argument and the number of loaded coordinates does not match the number of vertices. Is this a known shortcoming, or have we forgotten something to load periodic DMs? Best regards, Berend. On 9/22/21 20:59, Hapla Vaclav wrote: > To avoid confusions here, Berend seems to be specifically demanding XDMF > (PETSC_VIEWER_HDF5_XDMF). The stuff we are now working on is parallel > checkpointing in our own HDF5 format?(PETSC_VIEWER_HDF5_PETSC), I will > make a series of MRs on this topic in the following days. > > For XDMF, we are specifically missing the ability to write/load DMLabels > properly. XDMF uses specific cell-local numbering for faces for > specification of face sets, and face-local numbering for specification > of edge sets, which is not great wrt DMPlex design. And ParaView doesn't > show any of these properly so it's hard to debug. Matt, we should talk > about this soon. > > Berend, for now, could you just load the mesh initially from XDMF and > then use our PETSC_VIEWER_HDF5_PETSC format for subsequent saving/loading? > > Thanks, > > Vaclav > >> On 17 Sep 2021, at 15:46, Lawrence Mitchell > > wrote: >> >> Hi Berend, >> >>> On 14 Sep 2021, at 12:23, Matthew Knepley >> > wrote: >>> >>> On Tue, Sep 14, 2021 at 5:15 AM Berend van Wachem >>> > wrote: >>> Dear PETSc-team, >>> >>> We are trying to save and load distributed DMPlex and its associated >>> physical fields (created with DMCreateGlobalVector) ?(Uvelocity, >>> VVelocity, ?...) in HDF5_XDMF format. To achieve this, we do the >>> following: >>> >>> 1) save in the same xdmf.h5 file: >>> DMView( DM ????????, H5_XDMF_Viewer ); >>> VecView( UVelocity, H5_XDMF_Viewer ); >>> >>> 2) load the dm: >>> DMPlexCreateFromfile(PETSC_COMM_WORLD, Filename, PETSC_TRUE, DM); >>> >>> 3) load the physical field: >>> VecLoad( UVelocity, H5_XDMF_Viewer ); >>> >>> There are no errors in the execution, but the loaded DM is distributed >>> differently to the original one, which results in the incorrect >>> placement of the values of the physical fields (UVelocity etc.) in the >>> domain. >>> >>> This approach is used to restart the simulation with the last saved DM. >>> Is there something we are missing, or there exists alternative routes to >>> this goal? Can we somehow get the IS of the redistribution, so we can >>> re-distribute the vector data as well? >>> >>> Many thanks, best regards, >>> >>> Hi Berend, >>> >>> We are in the midst of rewriting this. We want to support saving >>> multiple meshes, with fields attached to each, >>> and preserving the discretization (section) information, and allowing >>> us to load up on a different number of >>> processes. We plan to be done by October. Vaclav and I are doing this >>> in collaboration with Koki Sagiyama, >>> David Ham, and Lawrence Mitchell from the Firedrake team. >> >> The core load/save cycle functionality is now in PETSc main. So if >> you're using main rather than a release, you can get access to it now. >> This section of the manual shows an example of how to do >> thingshttps://petsc.org/main/docs/manual/dmplex/#saving-and-loading-data-with-hdf5 >> >> >> Let us know if things aren't clear! >> >> Thanks, >> >> Lawrence > From f.levrero-florencio at onscale.com Wed Nov 17 11:15:48 2021 From: f.levrero-florencio at onscale.com (Francesc Levrero-Florencio) Date: Wed, 17 Nov 2021 17:15:48 +0000 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS In-Reply-To: <89B70D15-8949-45BF-B0D3-3B605AF84EA0@petsc.dev> References: <89B70D15-8949-45BF-B0D3-3B605AF84EA0@petsc.dev> Message-ID: Hi Barry, I believe that what you are referring to is what Jed is referring to in this thread, am I right? https://scicomp.stackexchange.com/questions/3298/appropriate-space-for-weak-solutions-to-an-elliptical-pde-with-mixed-inhomogeneo/3300#3300 We do set the rows/cols of the Jacobian to zero except the diagonal component which is set to one, as you mention. I understand that in the case of only homogeneous Dirichlet BCs it is generally a good idea to scale that diagonal component so that the condition number of the Jacobian improves. I assume that what Jed mentions is the inhomogeneous Dirichlet BC version of this scaling, which also acts on the corresponding indices of the residual, not just the Jacobian. My question is the following, since the case we are encountering problems with is a system with only homogeneous Dirichlet BCs, how does it apply? Also, would this scaling affect the convergence of the NEWTONLS with "bt" line-search? Without any scaling we can solve this example with "basic" (with a very reasonable convergence rate), but not with "bt" line-search. Regards, Francesc. On Tue, Nov 16, 2021 at 6:55 PM Barry Smith wrote: > > By "scaling" I mean you may need to rescale the residual terms from the > Dirichlet boundary conditions to be the same order (in terms of > the characteristic size of the elements) as they are for the other > residuals. > > It seems in your formulation the Jacobian entries of the residual for > Dirichlet boundary conditions are 1 on the diagonal while the Jacobian > entries of the residual for the PDE entries will have a scaling related to > the finite element method you are using, for example in 3d with an operator > grad dot grad the Jacobian entry will be order dx = (dx)^3 * (1/dx)^2. Thus > you should scale the Dirichlet boundary condition residuals by dx to get > the same scaling. > > Barry > > > On Nov 16, 2021, at 1:37 PM, Francesc Levrero-Florencio < > f.levrero-florencio at onscale.com> wrote: > > Hi Barry, > > Thanks for the quick reply. I am not sure what you meant by "scaling". In > our code, every time our residual is called we do the following in this > order: > - Apply updated Dirichlet BCs in order to assemble the internal forces, > and add them to the residual. > - Assemble the external forces, and subtract them from the residual. > - Update the residual to take into account Dirichet BCs, so we set the > corresponding indices of the residual to - (dirichlet_bc_value - > solution_value), the latter obtained directly from the solution vector of > PETSc. However, note that in this case all Dirichlet BCs are homogeneous, > so this value would be just "solution_value", or 0 throughout the > simulation. This ensures that the solution vector coming from PETSc has the > Dirichlet BCs applied. > > This is basically what we do in the residual. If I am not mistaken, the > lambda scaling factor from the line-search would scale the factor > "jacobian^{-1} * residual", so it would scale equally both internal and > external forces. > > Regards, > Francesc. > > On Tue, Nov 16, 2021 at 5:38 PM Barry Smith wrote: > >> >> Perhaps this behavior is the result of a "scaling" issue in how various >> terms affect the residual? In particular perhaps the terms for enforcing >> boundary conditions are scaled differently than terms for the PDE >> enforcement? >> >> >> >> > On Nov 16, 2021, at 11:19 AM, Francesc Levrero-Florencio < >> f.levrero-florencio at onscale.com> wrote: >> > >> > Dear PETSc team and users, >> > >> > We are running a simple cantilever beam bending, where the profile of >> the beam is I-shaped, where we apply a bending force on one end and fully >> constrained displacements on the other end. The formulation is a large >> strain formulation in Total Lagrangian form, where the material of the beam >> is a Saint Venant-Kirchhoff hyperelastic material that uses the same >> constants as steel (200E9 GPa Young?s modulus and 0.3 Poisson?s ratio). >> > >> > The simulation finishes in the requested number of time steps by using >> the ?basic? type of line-search in the SNES (i.e. traditional Newton method >> without line-search) in a reasonable number of Newton iterations per time >> step (3 or 4 iterations). However, when using the ?bt? (or ?l2?, and in >> fact no type of line-search ends up yielding convergence) line-search type, >> the convergence never happens within the SNES default maximum number of >> iterations of 50. >> > >> > During solving with traditional Newton, the general behaviour of each >> time step is that the norm of the residual increases on the second call to >> the residual function, but then hugely decreases in the following one or >> two, up to the point where convergence is achieved. Using ?bt? line-search, >> the line-search discards the step at lambda=1 immediately because the norm >> of the residual is higher than that produced in the first call to the >> residual function, cutting down the value of lambda to a value >> significantly lower than 1. The simulation then progresses in following >> Newton iterations in a similar fashion, the line-search step at lambda=1 is >> always discarded, and then smaller steps are taken but convergence never >> occurs, for even the first time step. >> > >> > Here are a few values of the relevant norms (using traditional Newton) >> in the first time step: >> > >> > BASIC NEWTON LS >> > Norm of the internal forces is 0 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1374.49 >> > Norm of the solution with Dirichlet BCs is 0 >> > Number of SNES iteration is 0 >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 113498 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 105053 >> > Norm of the solution with Dirichlet BCs is 0.441466 >> > Number of SNES iteration is 0 >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 42953.5 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 11.3734 >> > Norm of the solution with Dirichlet BCs is 0.441438 >> > Number of SNES iteration is 1 >> > >> > Here are a few values of the relevant norms (using ?bt? line-search) in >> the first time step: >> > >> > BT NEWTONLS >> > Norm of the internal forces is 0 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1374.49 >> > Norm of the solution with Dirichlet BCs is 0 >> > Number of SNES iteration is 0 >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 113498 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 105053 >> > Norm of the solution with Dirichlet BCs is 0.441466 >> > Number of SNES iteration is 0 >> > (I assume this is the first try at lambda=1) >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 4422.12 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1622.74 >> > Norm of the solution with Dirichlet BCs is 0.0441466 >> > Number of SNES iteration is 0 >> > Line search: gnorm after quadratic fit 1.622742343614e+03 >> > (I assume that in this line-search iteration 0.05 < lambda < 1, but the >> corresponding residual is not smaller than the one in the first call, 1622 >> > 1374) >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 2163.76 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1331.88 >> > Norm of the solution with Dirichlet BCs is 0.0220733 >> > Number of SNES iteration is 0 >> > Line search: Cubically determined step, current gnorm >> 1.331884625811e+03 lambda=5.0000000000000003e-02 >> > (This is the accepted lambda for the current Newton iteration) >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 104020 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 94739 >> > Norm of the solution with Dirichlet BCs is 0.441323 >> > Number of SNES iteration is 1 >> > >> > My question would be, any idea on how to deal with this situation? I >> would imagine a ?hack? would be to bypass the first residual norm, and have >> the line-search use the following one as the ?base norm? to check its >> reduction in further iterations, but we are open to any ideas. >> > >> > Thanks for your help in advance and please keep up the good work! >> > >> > Regards, >> > Francesc. >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jed at jedbrown.org Wed Nov 17 11:49:25 2021 From: jed at jedbrown.org (Jed Brown) Date: Wed, 17 Nov 2021 10:49:25 -0700 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS In-Reply-To: References: <89B70D15-8949-45BF-B0D3-3B605AF84EA0@petsc.dev> Message-ID: <87tugar7tm.fsf@jedbrown.org> Francesc Levrero-Florencio writes: > Hi Barry, > > I believe that what you are referring to is what Jed is referring to in > this thread, am I right? > https://scicomp.stackexchange.com/questions/3298/appropriate-space-for-weak-solutions-to-an-elliptical-pde-with-mixed-inhomogeneo/3300#3300 Yeah, that's the scaling. Are you decoupling the interior in the way I described, so the matrix columns for essential boundary conditions are also zeroed? Also note that line searches can prevent a rootfinding method from converging, as in this example. https://scicomp.stackexchange.com/a/2446/119 There is -snes_linesearch_type cp ("critical point"), which has a surrogate that looks like aWolfe conditions when your rootfinding problem happens to be the first order optimality conditions for a minimization problem. There's also SNESSetObjective(), if your problem has an explicit objective. In practice, cp usually works well if your problem is "almost" coming from a minimization principle, and poorly otherwise. > We do set the rows/cols of the Jacobian to zero except the diagonal > component which is set to one, as you mention. I understand that in the > case of only homogeneous Dirichlet BCs it is generally a good idea to scale > that diagonal component so that the condition number of the Jacobian > improves. I assume that what Jed mentions is the inhomogeneous Dirichlet BC > version of this scaling, which also acts on the corresponding indices of > the residual, not just the Jacobian. My question is the following, since > the case we are encountering problems with is a system with only > homogeneous Dirichlet BCs, how does it apply? Also, would this scaling > affect the convergence of the NEWTONLS with "bt" line-search? Without any > scaling we can solve this example with "basic" (with a very reasonable > convergence rate), but not with "bt" line-search. From bsmith at petsc.dev Wed Nov 17 14:27:48 2021 From: bsmith at petsc.dev (Barry Smith) Date: Wed, 17 Nov 2021 15:27:48 -0500 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS In-Reply-To: References: <89B70D15-8949-45BF-B0D3-3B605AF84EA0@petsc.dev> Message-ID: <5EED810A-E084-4C44-A09E-9FE021C7891C@petsc.dev> I do not know if the scaling of the BC equations could affect the norms of the nonlinear residual in your situation. I suggest you look at which coefficients of the residual are blowing up in the first step to see if you can understand why they get increased. Barry > On Nov 17, 2021, at 12:15 PM, Francesc Levrero-Florencio wrote: > > Hi Barry, > > I believe that what you are referring to is what Jed is referring to in this thread, am I right? > https://scicomp.stackexchange.com/questions/3298/appropriate-space-for-weak-solutions-to-an-elliptical-pde-with-mixed-inhomogeneo/3300#3300 > > We do set the rows/cols of the Jacobian to zero except the diagonal component which is set to one, as you mention. I understand that in the case of only homogeneous Dirichlet BCs it is generally a good idea to scale that diagonal component so that the condition number of the Jacobian improves. I assume that what Jed mentions is the inhomogeneous Dirichlet BC version of this scaling, which also acts on the corresponding indices of the residual, not just the Jacobian. My question is the following, since the case we are encountering problems with is a system with only homogeneous Dirichlet BCs, how does it apply? Also, would this scaling affect the convergence of the NEWTONLS with "bt" line-search? Without any scaling we can solve this example with "basic" (with a very reasonable convergence rate), but not with "bt" line-search. > > Regards, > Francesc. > > On Tue, Nov 16, 2021 at 6:55 PM Barry Smith > wrote: > > By "scaling" I mean you may need to rescale the residual terms from the Dirichlet boundary conditions to be the same order (in terms of the characteristic size of the elements) as they are for the other residuals. > > It seems in your formulation the Jacobian entries of the residual for Dirichlet boundary conditions are 1 on the diagonal while the Jacobian entries of the residual for the PDE entries will have a scaling related to the finite element method you are using, for example in 3d with an operator grad dot grad the Jacobian entry will be order dx = (dx)^3 * (1/dx)^2. Thus you should scale the Dirichlet boundary condition residuals by dx to get the same scaling. > > Barry > > >> On Nov 16, 2021, at 1:37 PM, Francesc Levrero-Florencio > wrote: >> >> Hi Barry, >> >> Thanks for the quick reply. I am not sure what you meant by "scaling". In our code, every time our residual is called we do the following in this order: >> - Apply updated Dirichlet BCs in order to assemble the internal forces, and add them to the residual. >> - Assemble the external forces, and subtract them from the residual. >> - Update the residual to take into account Dirichet BCs, so we set the corresponding indices of the residual to - (dirichlet_bc_value - solution_value), the latter obtained directly from the solution vector of PETSc. However, note that in this case all Dirichlet BCs are homogeneous, so this value would be just "solution_value", or 0 throughout the simulation. This ensures that the solution vector coming from PETSc has the Dirichlet BCs applied. >> >> This is basically what we do in the residual. If I am not mistaken, the lambda scaling factor from the line-search would scale the factor "jacobian^{-1} * residual", so it would scale equally both internal and external forces. >> >> Regards, >> Francesc. >> >> On Tue, Nov 16, 2021 at 5:38 PM Barry Smith > wrote: >> >> Perhaps this behavior is the result of a "scaling" issue in how various terms affect the residual? In particular perhaps the terms for enforcing boundary conditions are scaled differently than terms for the PDE enforcement? >> >> >> >> > On Nov 16, 2021, at 11:19 AM, Francesc Levrero-Florencio > wrote: >> > >> > Dear PETSc team and users, >> > >> > We are running a simple cantilever beam bending, where the profile of the beam is I-shaped, where we apply a bending force on one end and fully constrained displacements on the other end. The formulation is a large strain formulation in Total Lagrangian form, where the material of the beam is a Saint Venant-Kirchhoff hyperelastic material that uses the same constants as steel (200E9 GPa Young?s modulus and 0.3 Poisson?s ratio). >> > >> > The simulation finishes in the requested number of time steps by using the ?basic? type of line-search in the SNES (i.e. traditional Newton method without line-search) in a reasonable number of Newton iterations per time step (3 or 4 iterations). However, when using the ?bt? (or ?l2?, and in fact no type of line-search ends up yielding convergence) line-search type, the convergence never happens within the SNES default maximum number of iterations of 50. >> > >> > During solving with traditional Newton, the general behaviour of each time step is that the norm of the residual increases on the second call to the residual function, but then hugely decreases in the following one or two, up to the point where convergence is achieved. Using ?bt? line-search, the line-search discards the step at lambda=1 immediately because the norm of the residual is higher than that produced in the first call to the residual function, cutting down the value of lambda to a value significantly lower than 1. The simulation then progresses in following Newton iterations in a similar fashion, the line-search step at lambda=1 is always discarded, and then smaller steps are taken but convergence never occurs, for even the first time step. >> > >> > Here are a few values of the relevant norms (using traditional Newton) in the first time step: >> > >> > BASIC NEWTON LS >> > Norm of the internal forces is 0 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1374.49 >> > Norm of the solution with Dirichlet BCs is 0 >> > Number of SNES iteration is 0 >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 113498 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 105053 >> > Norm of the solution with Dirichlet BCs is 0.441466 >> > Number of SNES iteration is 0 >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 42953.5 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 11.3734 >> > Norm of the solution with Dirichlet BCs is 0.441438 >> > Number of SNES iteration is 1 >> > >> > Here are a few values of the relevant norms (using ?bt? line-search) in the first time step: >> > >> > BT NEWTONLS >> > Norm of the internal forces is 0 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1374.49 >> > Norm of the solution with Dirichlet BCs is 0 >> > Number of SNES iteration is 0 >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 113498 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 105053 >> > Norm of the solution with Dirichlet BCs is 0.441466 >> > Number of SNES iteration is 0 >> > (I assume this is the first try at lambda=1) >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 4422.12 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1622.74 >> > Norm of the solution with Dirichlet BCs is 0.0441466 >> > Number of SNES iteration is 0 >> > Line search: gnorm after quadratic fit 1.622742343614e+03 >> > (I assume that in this line-search iteration 0.05 < lambda < 1, but the corresponding residual is not smaller than the one in the first call, 1622 > 1374) >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 2163.76 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 1331.88 >> > Norm of the solution with Dirichlet BCs is 0.0220733 >> > Number of SNES iteration is 0 >> > Line search: Cubically determined step, current gnorm 1.331884625811e+03 lambda=5.0000000000000003e-02 >> > (This is the accepted lambda for the current Newton iteration) >> > --------------------------------------------------------------------- >> > Norm of the internal forces is 104020 >> > Norm of the external forces is 1374.49 >> > Norm of the residual is 94739 >> > Norm of the solution with Dirichlet BCs is 0.441323 >> > Number of SNES iteration is 1 >> > >> > My question would be, any idea on how to deal with this situation? I would imagine a ?hack? would be to bypass the first residual norm, and have the line-search use the following one as the ?base norm? to check its reduction in further iterations, but we are open to any ideas. >> > >> > Thanks for your help in advance and please keep up the good work! >> > >> > Regards, >> > Francesc. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From f.levrero-florencio at onscale.com Wed Nov 17 15:17:36 2021 From: f.levrero-florencio at onscale.com (Francesc Levrero-Florencio) Date: Wed, 17 Nov 2021 21:17:36 +0000 Subject: [petsc-users] Doubt about BT and BASIC NEWTONLS In-Reply-To: <87tugar7tm.fsf@jedbrown.org> References: <89B70D15-8949-45BF-B0D3-3B605AF84EA0@petsc.dev> <87tugar7tm.fsf@jedbrown.org> Message-ID: Hi Jed, Thanks for the information. Yes, we set the rows and cols of the matrix for essential BCs to zero (except diagonal, to 1). For homogeneous BCs it would be exactly as the first term you mentioned in the link, and for nonhomogeneous BCs it would be both terms, but the second one not scale (i.e. alpha set to 1). I have tried a few of the line searches, "bt" does not work, but "cp" does progress for a few time steps, with reasonable numbers of Newton iterations (ends up stalling though). It does look to me that the "bt" line search seems to "get stuck" on a local minima, while the full Newton, maybe due to the size of the step, gets to overcome it. I am just a bit "surprised" because in my experience line searches, particularly of the "bt" kind, tend to improve, or at least not decrease, the convergence behaviour of the Newton solver. Regards, Francesc. On Wed, Nov 17, 2021 at 5:49 PM Jed Brown wrote: > Francesc Levrero-Florencio writes: > > > Hi Barry, > > > > I believe that what you are referring to is what Jed is referring to in > > this thread, am I right? > > > https://scicomp.stackexchange.com/questions/3298/appropriate-space-for-weak-solutions-to-an-elliptical-pde-with-mixed-inhomogeneo/3300#3300 > > Yeah, that's the scaling. Are you decoupling the interior in the way I > described, so the matrix columns for essential boundary conditions are also > zeroed? > > Also note that line searches can prevent a rootfinding method from > converging, as in this example. > > https://scicomp.stackexchange.com/a/2446/119 > > There is -snes_linesearch_type cp ("critical point"), which has a > surrogate that looks like aWolfe conditions when your rootfinding problem > happens to be the first order optimality conditions for a minimization > problem. There's also SNESSetObjective(), if your problem has an explicit > objective. In practice, cp usually works well if your problem is "almost" > coming from a minimization principle, and poorly otherwise. > > > We do set the rows/cols of the Jacobian to zero except the diagonal > > component which is set to one, as you mention. I understand that in the > > case of only homogeneous Dirichlet BCs it is generally a good idea to > scale > > that diagonal component so that the condition number of the Jacobian > > improves. I assume that what Jed mentions is the inhomogeneous Dirichlet > BC > > version of this scaling, which also acts on the corresponding indices of > > the residual, not just the Jacobian. My question is the following, since > > the case we are encountering problems with is a system with only > > homogeneous Dirichlet BCs, how does it apply? Also, would this scaling > > affect the convergence of the NEWTONLS with "bt" line-search? Without any > > scaling we can solve this example with "basic" (with a very reasonable > > convergence rate), but not with "bt" line-search. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k.sagiyama at imperial.ac.uk Thu Nov 18 17:26:14 2021 From: k.sagiyama at imperial.ac.uk (Sagiyama, Koki) Date: Thu, 18 Nov 2021 23:26:14 +0000 Subject: [petsc-users] DMView and DMLoad In-Reply-To: <6c4e0656-db99-e9da-000f-ab9f7dd62c07@ovgu.de> References: <56ce2135-9757-4292-e33b-c7eea8fb7b2e@ovgu.de> <056E066F-D596-4254-A44A-60BFFD30FE82@erdw.ethz.ch> <6c4e0656-db99-e9da-000f-ab9f7dd62c07@ovgu.de> Message-ID: Hi Berend, I was not able to reproduce the issue you are having, but the following 1D example (and similar 2D examples) worked fine for me using the latest PETSc. Please note that DMPlexCoordinatesLoad() now takes a PetscSF object as the third argument, but the default behavior is unchanged. /* test_periodic_io.c */ #include #include #include int main(int argc, char **argv) { DM dm; Vec coordinates; PetscViewer viewer; PetscViewerFormat format = PETSC_VIEWER_HDF5_PETSC; PetscSF sfO; PetscErrorCode ierr; ierr = PetscInitialize(&argc, &argv, NULL, NULL); if (ierr) return ierr; /* Save */ ierr = PetscViewerHDF5Open(PETSC_COMM_WORLD, "periodic_example.h5", FILE_MODE_WRITE, &viewer);CHKERRQ(ierr); { DM pdm; PetscInt dim = 1; const PetscInt faces[1] = {4}; DMBoundaryType periodicity[] = {DM_BOUNDARY_PERIODIC}; PetscInt overlap = 1; ierr = DMPlexCreateBoxMesh(PETSC_COMM_WORLD, dim, PETSC_FALSE, faces, NULL, NULL, periodicity, PETSC_TRUE, &dm);CHKERRQ(ierr); ierr = DMPlexDistribute(dm, overlap, NULL, &pdm);CHKERRQ(ierr); ierr = DMDestroy(&dm);CHKERRQ(ierr); dm = pdm; ierr = PetscObjectSetName((PetscObject)dm, "periodicDM");CHKERRQ(ierr); } ierr = DMGetCoordinates(dm, &coordinates);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD, "Coordinates before saving:\n");CHKERRQ(ierr); ierr = VecView(coordinates, NULL);CHKERRQ(ierr); ierr = PetscViewerPushFormat(viewer, format);CHKERRQ(ierr); ierr = DMPlexTopologyView(dm, viewer);CHKERRQ(ierr); ierr = DMPlexCoordinatesView(dm, viewer);CHKERRQ(ierr); ierr = PetscViewerPopFormat(viewer);CHKERRQ(ierr); ierr = DMDestroy(&dm);CHKERRQ(ierr); ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr); /* Load */ ierr = PetscViewerHDF5Open(PETSC_COMM_WORLD, "periodic_example.h5", FILE_MODE_READ, &viewer);CHKERRQ(ierr); ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); ierr = PetscObjectSetName((PetscObject)dm, "periodicDM");CHKERRQ(ierr); ierr = PetscViewerPushFormat(viewer, format);CHKERRQ(ierr); ierr = DMPlexTopologyLoad(dm, viewer, &sfO);CHKERRQ(ierr); ierr = DMPlexCoordinatesLoad(dm, viewer, sfO);CHKERRQ(ierr); ierr = PetscViewerPopFormat(viewer);CHKERRQ(ierr); ierr = DMGetCoordinates(dm, &coordinates);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD, "Coordinates after loading:\n");CHKERRQ(ierr); ierr = VecView(coordinates, NULL);CHKERRQ(ierr); ierr = PetscSFDestroy(&sfO);CHKERRQ(ierr); ierr = DMDestroy(&dm);CHKERRQ(ierr); ierr = PetscViewerDestroy(&viewer);CHKERRQ(ierr); ierr = PetscFinalize(); return ierr; } mpiexec -n 2 ./test_periodic_io Coordinates before saving: Vec Object: coordinates 2 MPI processes type: mpi Process [0] 0. Process [1] 0.25 0.5 0.75 Coordinates after loading: Vec Object: vertices 2 MPI processes type: mpi Process [0] 0. 0.25 0.5 0.75 Process [1] I would also like to note that, with the latest update, we can optionally load coordinates directly on the distributed dm as (using your notation): /* Distribute dm */ ... PetscSFCompose(sfO, sfDist, &sf); DMPlexCoordinatesLoad(dm, viewer, sf); To use this feature, we need to pass "-dm_plex_view_hdf5_storage_version 2.0.0" option when saving topology/coordinates. Thanks, Koki ________________________________ From: Berend van Wachem Sent: Wednesday, November 17, 2021 3:16 PM To: Hapla Vaclav ; PETSc users list ; Lawrence Mitchell ; Sagiyama, Koki Subject: Re: [petsc-users] DMView and DMLoad ******************* This email originates from outside Imperial. Do not click on links and attachments unless you recognise the sender. If you trust the sender, add them to your safe senders list https://spam.ic.ac.uk/SpamConsole/Senders.aspx to disable email stamping for this address. ******************* Dear Vaclav, Lawrence, Koki, Thanks for your help! Following your advice and following your example (https://petsc.org/main/docs/manual/dmplex/#saving-and-loading-data-with-hdf5) we are able to save and load the DM with a wrapped Vector in h5 format (PETSC_VIEWER_HDF5_PETSC) successfully. For saving, we use something similar to: DMPlexTopologyView(dm, viewer); DMClone(dm, &sdm); ... DMPlexSectionView(dm, viewer, sdm); DMGetLocalVector(sdm, &vec); ... DMPlexLocalVectorView(dm, viewer, sdm, vec); and for loading: DMCreate(PETSC_COMM_WORLD, &dm); DMSetType(dm, DMPLEX); ... PetscViewerPushFormat(viewer, PETSC_VIEWER_HDF5_PETSC); DMPlexTopologyLoad(dm, viewer, &sfO); DMPlexLabelsLoad(dm, viewer); DMPlexCoordinatesLoad(dm, viewer); PetscViewerPopFormat(viewer); ... PetscSFCompose(sfO, sfDist, &sf); ... DMClone(dm, &sdm); DMPlexSectionLoad(dm, viewer, sdm, sf, &globalDataSF, &localDataSF); DMGetLocalVector(sdm, &vec); ... DMPlexLocalVectorLoad(dm, viewer, sdm, localDataSF, vec); This works fine for non-periodic DMs but for periodic cases the line: DMPlexCoordinatesLoad(dm, H5Viewer); delivers the error message: invalid argument and the number of loaded coordinates does not match the number of vertices. Is this a known shortcoming, or have we forgotten something to load periodic DMs? Best regards, Berend. On 9/22/21 20:59, Hapla Vaclav wrote: > To avoid confusions here, Berend seems to be specifically demanding XDMF > (PETSC_VIEWER_HDF5_XDMF). The stuff we are now working on is parallel > checkpointing in our own HDF5 format (PETSC_VIEWER_HDF5_PETSC), I will > make a series of MRs on this topic in the following days. > > For XDMF, we are specifically missing the ability to write/load DMLabels > properly. XDMF uses specific cell-local numbering for faces for > specification of face sets, and face-local numbering for specification > of edge sets, which is not great wrt DMPlex design. And ParaView doesn't > show any of these properly so it's hard to debug. Matt, we should talk > about this soon. > > Berend, for now, could you just load the mesh initially from XDMF and > then use our PETSC_VIEWER_HDF5_PETSC format for subsequent saving/loading? > > Thanks, > > Vaclav > >> On 17 Sep 2021, at 15:46, Lawrence Mitchell > > wrote: >> >> Hi Berend, >> >>> On 14 Sep 2021, at 12:23, Matthew Knepley >> > wrote: >>> >>> On Tue, Sep 14, 2021 at 5:15 AM Berend van Wachem >>> > wrote: >>> Dear PETSc-team, >>> >>> We are trying to save and load distributed DMPlex and its associated >>> physical fields (created with DMCreateGlobalVector) (Uvelocity, >>> VVelocity, ...) in HDF5_XDMF format. To achieve this, we do the >>> following: >>> >>> 1) save in the same xdmf.h5 file: >>> DMView( DM , H5_XDMF_Viewer ); >>> VecView( UVelocity, H5_XDMF_Viewer ); >>> >>> 2) load the dm: >>> DMPlexCreateFromfile(PETSC_COMM_WORLD, Filename, PETSC_TRUE, DM); >>> >>> 3) load the physical field: >>> VecLoad( UVelocity, H5_XDMF_Viewer ); >>> >>> There are no errors in the execution, but the loaded DM is distributed >>> differently to the original one, which results in the incorrect >>> placement of the values of the physical fields (UVelocity etc.) in the >>> domain. >>> >>> This approach is used to restart the simulation with the last saved DM. >>> Is there something we are missing, or there exists alternative routes to >>> this goal? Can we somehow get the IS of the redistribution, so we can >>> re-distribute the vector data as well? >>> >>> Many thanks, best regards, >>> >>> Hi Berend, >>> >>> We are in the midst of rewriting this. We want to support saving >>> multiple meshes, with fields attached to each, >>> and preserving the discretization (section) information, and allowing >>> us to load up on a different number of >>> processes. We plan to be done by October. Vaclav and I are doing this >>> in collaboration with Koki Sagiyama, >>> David Ham, and Lawrence Mitchell from the Firedrake team. >> >> The core load/save cycle functionality is now in PETSc main. So if >> you're using main rather than a release, you can get access to it now. >> This section of the manual shows an example of how to do >> thingshttps://petsc.org/main/docs/manual/dmplex/#saving-and-loading-data-with-hdf5 >> >> >> Let us know if things aren't clear! >> >> Thanks, >> >> Lawrence > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sam.guo at cd-adapco.com Thu Nov 18 23:30:19 2021 From: sam.guo at cd-adapco.com (Sam Guo) Date: Thu, 18 Nov 2021 21:30:19 -0800 Subject: [petsc-users] PCSHELL does not support getting factor matrix Message-ID: Dear PETSc dev team, I am implementing SLEPc interval option using shell matrix as follows: EPSGetST(eps, &st); STSetType(st, STSINVERT); STGetKSP(st, &ksp); KSPSetOperators(ksp, A, A); KSPSetType(ksp, KSPPREONLY); KSPGetPC(ksp, &pc); KSPGetPC(ksp, &pc) MatSetOption(A, MAT_SPD, PETSC_TRUE); PCSetType(pc, PCSHELL); PCShellSetContext(pc, &appCtx); PCShellSetApply(pc, applyPreconditioner); PetscObjectComposeFunction((PetscObject)pc,"PCFactorGetZeroPivot_C",PCFactorGetZeroPivot_C); When I run it, I get the following error. Any idea what I did wrong? Thanks a lot for your help. [0]PETSC ERROR: No support for this operation for this object type [0]PETSC ERROR: PC type does not support getting factor matrix [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 [0]PETSC ERROR: Unknown Name on a arch-starccmplus_serial_real named pl2usbvu0037pc.net.plm.eds.com by cd4hhv Thu Nov 18 21:11:45 2021 Number of iterations of the method: 0 [0]PETSC ERROR: Configure options --with-x=0 --with-fc=0 --with-debugging=1 --with-blaslapack-dir=/u/cd4hhv/dev2/mkl/2017.2-cda-001/linux/lib/intel64/../.. --with-mpi=0 -CFLAGS=-g -CXXFLAGS=-g --with-clean=1 --force --with-scalar-type=real [0]PETSC ERROR: #1 PCFactorGetMatrix() line 1332 in ../../../petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #2 EPSSliceGetInertia() line 340 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [0]PETSC ERROR: #3 EPSSetUp_KrylovSchur_Slice() line 467 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [0]PETSC ERROR: #4 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c [0]PETSC ERROR: #5 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c Solution method: krylovschur [0]PETSC ERROR: #6 EPSSliceGetEPS() line 306 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [0]PETSC ERROR: #7 EPSSetUp_KrylovSchur_Slice() line 416 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c [0]PETSC ERROR: #8 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c [0]PETSC ERROR: #9 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Thu Nov 18 23:34:03 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Fri, 19 Nov 2021 14:34:03 +0900 Subject: [petsc-users] How to check NULL pointer in Fortran Message-ID: Dear PETSc-team, I am using function PetscSFGetGraph in my program like ----------------------- call PetscSFGetGraph(sf,gnroots,gnleaves,gmine,gremote,ierr) ----------------------- In some cases, it works well. But in some cases, I encountered following error ------------------ PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range ------------------ I found it is due to the action of reading gmine, which is a fortran pointer point to an array. It is reasonable because PETSc manual tells me "if returned value is NULL, it means leaves are in contiguous storage". The problem is that I cannot find if gmine is a null pointer. 1) I cannot use "if (gmine==PETSC_NULL_INTEGER)" because my intel compiler would return a compile error "A scalar-valued expression is required in this context" 2) When using standard style of checking a null pointer in fortran, "associated(gmine)", it returns "T". Even in cases such action of "print *, gmine(1)" would give rise to above Segmentation Violation error. Is there any means to check the NULL pointer in Fortran in above cases. Many thanks, Yuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Fri Nov 19 03:41:15 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 19 Nov 2021 10:41:15 +0100 Subject: [petsc-users] PCSHELL does not support getting factor matrix In-Reply-To: References: Message-ID: <108CFB57-A068-4ADF-9E75-BF0876D77174@dsic.upv.es> It is trying to call PCFactorGetMatrix() on your PC, but this operation is not supported by PCSHELL and it is not possible to set it via PetscObjectComposeFunction(). PCSHELL uses the PCShellSet* interface, that is restricted to a limited number of operations. Jose > El 19 nov 2021, a las 6:30, Sam Guo escribi?: > > Dear PETSc dev team, > I am implementing SLEPc interval option using shell matrix as follows: > > EPSGetST(eps, &st); > STSetType(st, STSINVERT); > STGetKSP(st, &ksp); > KSPSetOperators(ksp, A, A); > KSPSetType(ksp, KSPPREONLY); > KSPGetPC(ksp, &pc); > KSPGetPC(ksp, &pc) > MatSetOption(A, MAT_SPD, PETSC_TRUE); > PCSetType(pc, PCSHELL); > PCShellSetContext(pc, &appCtx); > PCShellSetApply(pc, applyPreconditioner); > PetscObjectComposeFunction((PetscObject)pc,"PCFactorGetZeroPivot_C",PCFactorGetZeroPivot_C); > > When I run it, I get the following error. Any idea what I did wrong? Thanks a lot for your help. > > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: PC type does not support getting factor matrix > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 > [0]PETSC ERROR: Unknown Name on a arch-starccmplus_serial_real named pl2usbvu0037pc.net.plm.eds.com by cd4hhv Thu Nov 18 21:11:45 2021 > Number of iterations of the method: 0 > [0]PETSC ERROR: Configure options --with-x=0 --with-fc=0 --with-debugging=1 --with-blaslapack-dir=/u/cd4hhv/dev2/mkl/2017.2-cda-001/linux/lib/intel64/../.. --with-mpi=0 -CFLAGS=-g -CXXFLAGS=-g --with-clean=1 --force --with-scalar-type=real > [0]PETSC ERROR: #1 PCFactorGetMatrix() line 1332 in ../../../petsc/src/ksp/pc/interface/precon.c > [0]PETSC ERROR: #2 EPSSliceGetInertia() line 340 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [0]PETSC ERROR: #3 EPSSetUp_KrylovSchur_Slice() line 467 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [0]PETSC ERROR: #4 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > [0]PETSC ERROR: #5 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c > Solution method: krylovschur > [0]PETSC ERROR: #6 EPSSliceGetEPS() line 306 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [0]PETSC ERROR: #7 EPSSetUp_KrylovSchur_Slice() line 416 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c > [0]PETSC ERROR: #8 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c > [0]PETSC ERROR: #9 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c From mfadams at lbl.gov Fri Nov 19 06:40:37 2021 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 19 Nov 2021 07:40:37 -0500 Subject: [petsc-users] How to check NULL pointer in Fortran In-Reply-To: References: Message-ID: You could try this: integer :: i real :: r i = loc(r) print *, i On Fri, Nov 19, 2021 at 12:34 AM ?? wrote: > Dear PETSc-team, > > I am using function PetscSFGetGraph in my program like > ----------------------- > call PetscSFGetGraph(sf,gnroots,gnleaves,gmine,gremote,ierr) > ----------------------- > > In some cases, it works well. But in some cases, I encountered following error > ------------------ > PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > > ------------------ > > I found it is due to the action of reading gmine, which is a fortran pointer point to an array. It is reasonable because PETSc manual tells me "if returned value is NULL, it means leaves are in contiguous storage". The problem is that I cannot find if gmine is a null pointer. > > 1) I cannot use "if (gmine==PETSC_NULL_INTEGER)" because my intel compiler would return a compile error "A scalar-valued expression is required in this context" > > 2) When using standard style of checking a null pointer in fortran, "associated(gmine)", it returns "T". Even in cases such action of "print *, gmine(1)" would give rise to above Segmentation Violation error. > > Is there any means to check the NULL pointer in Fortran in above cases. > > Many thanks, > > Yuan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Fri Nov 19 06:45:11 2021 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 19 Nov 2021 15:45:11 +0300 Subject: [petsc-users] PCSHELL does not support getting factor matrix In-Reply-To: <108CFB57-A068-4ADF-9E75-BF0876D77174@dsic.upv.es> References: <108CFB57-A068-4ADF-9E75-BF0876D77174@dsic.upv.es> Message-ID: <18BFAFD7-ECDE-4096-BC98-2343326449C8@gmail.com> Jose Now that we have the PCMatApply interface, you could switch to use that inside SLEPc. I guess you are using MatSolve, right? If not, the alternative is to have a PCFactorGetMatrix with creates on the fly an object that behaves like it. The problem is that we do not have a matching restore, and the new object created will be leaked. What do you think? > On Nov 19, 2021, at 12:41 PM, Jose E. Roman wrote: > > It is trying to call PCFactorGetMatrix() on your PC, but this operation is not supported by PCSHELL and it is not possible to set it via PetscObjectComposeFunction(). PCSHELL uses the PCShellSet* interface, that is restricted to a limited number of operations. > > Jose > > >> El 19 nov 2021, a las 6:30, Sam Guo escribi?: >> >> Dear PETSc dev team, >> I am implementing SLEPc interval option using shell matrix as follows: >> >> EPSGetST(eps, &st); >> STSetType(st, STSINVERT); >> STGetKSP(st, &ksp); >> KSPSetOperators(ksp, A, A); >> KSPSetType(ksp, KSPPREONLY); >> KSPGetPC(ksp, &pc); >> KSPGetPC(ksp, &pc) >> MatSetOption(A, MAT_SPD, PETSC_TRUE); >> PCSetType(pc, PCSHELL); >> PCShellSetContext(pc, &appCtx); >> PCShellSetApply(pc, applyPreconditioner); >> PetscObjectComposeFunction((PetscObject)pc,"PCFactorGetZeroPivot_C",PCFactorGetZeroPivot_C); >> >> When I run it, I get the following error. Any idea what I did wrong? Thanks a lot for your help. >> >> [0]PETSC ERROR: No support for this operation for this object type >> [0]PETSC ERROR: PC type does not support getting factor matrix >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 >> [0]PETSC ERROR: Unknown Name on a arch-starccmplus_serial_real named pl2usbvu0037pc.net.plm.eds.com by cd4hhv Thu Nov 18 21:11:45 2021 >> Number of iterations of the method: 0 >> [0]PETSC ERROR: Configure options --with-x=0 --with-fc=0 --with-debugging=1 --with-blaslapack-dir=/u/cd4hhv/dev2/mkl/2017.2-cda-001/linux/lib/intel64/../.. --with-mpi=0 -CFLAGS=-g -CXXFLAGS=-g --with-clean=1 --force --with-scalar-type=real >> [0]PETSC ERROR: #1 PCFactorGetMatrix() line 1332 in ../../../petsc/src/ksp/pc/interface/precon.c >> [0]PETSC ERROR: #2 EPSSliceGetInertia() line 340 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> [0]PETSC ERROR: #3 EPSSetUp_KrylovSchur_Slice() line 467 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> [0]PETSC ERROR: #4 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >> [0]PETSC ERROR: #5 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c >> Solution method: krylovschur >> [0]PETSC ERROR: #6 EPSSliceGetEPS() line 306 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> [0]PETSC ERROR: #7 EPSSetUp_KrylovSchur_Slice() line 416 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >> [0]PETSC ERROR: #8 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >> [0]PETSC ERROR: #9 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c > From stefano.zampini at gmail.com Fri Nov 19 06:46:05 2021 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Fri, 19 Nov 2021 15:46:05 +0300 Subject: [petsc-users] PCSHELL does not support getting factor matrix In-Reply-To: <18BFAFD7-ECDE-4096-BC98-2343326449C8@gmail.com> References: <108CFB57-A068-4ADF-9E75-BF0876D77174@dsic.upv.es> <18BFAFD7-ECDE-4096-BC98-2343326449C8@gmail.com> Message-ID: <7A8AE4DA-5CAB-48BC-AAF5-B2F8D24715F1@gmail.com> Oh, I see the stack trace now. This requires to compute the inertia? > On Nov 19, 2021, at 3:45 PM, Stefano Zampini wrote: > > Jose > > Now that we have the PCMatApply interface, you could switch to use that inside SLEPc. I guess you are using MatSolve, right? > If not, the alternative is to have a PCFactorGetMatrix with creates on the fly an object that behaves like it. The problem is that we do not have a matching restore, and the new object created will be leaked. > What do you think? > > >> On Nov 19, 2021, at 12:41 PM, Jose E. Roman wrote: >> >> It is trying to call PCFactorGetMatrix() on your PC, but this operation is not supported by PCSHELL and it is not possible to set it via PetscObjectComposeFunction(). PCSHELL uses the PCShellSet* interface, that is restricted to a limited number of operations. >> >> Jose >> >> >>> El 19 nov 2021, a las 6:30, Sam Guo escribi?: >>> >>> Dear PETSc dev team, >>> I am implementing SLEPc interval option using shell matrix as follows: >>> >>> EPSGetST(eps, &st); >>> STSetType(st, STSINVERT); >>> STGetKSP(st, &ksp); >>> KSPSetOperators(ksp, A, A); >>> KSPSetType(ksp, KSPPREONLY); >>> KSPGetPC(ksp, &pc); >>> KSPGetPC(ksp, &pc) >>> MatSetOption(A, MAT_SPD, PETSC_TRUE); >>> PCSetType(pc, PCSHELL); >>> PCShellSetContext(pc, &appCtx); >>> PCShellSetApply(pc, applyPreconditioner); >>> PetscObjectComposeFunction((PetscObject)pc,"PCFactorGetZeroPivot_C",PCFactorGetZeroPivot_C); >>> >>> When I run it, I get the following error. Any idea what I did wrong? Thanks a lot for your help. >>> >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: PC type does not support getting factor matrix >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 >>> [0]PETSC ERROR: Unknown Name on a arch-starccmplus_serial_real named pl2usbvu0037pc.net.plm.eds.com by cd4hhv Thu Nov 18 21:11:45 2021 >>> Number of iterations of the method: 0 >>> [0]PETSC ERROR: Configure options --with-x=0 --with-fc=0 --with-debugging=1 --with-blaslapack-dir=/u/cd4hhv/dev2/mkl/2017.2-cda-001/linux/lib/intel64/../.. --with-mpi=0 -CFLAGS=-g -CXXFLAGS=-g --with-clean=1 --force --with-scalar-type=real >>> [0]PETSC ERROR: #1 PCFactorGetMatrix() line 1332 in ../../../petsc/src/ksp/pc/interface/precon.c >>> [0]PETSC ERROR: #2 EPSSliceGetInertia() line 340 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> [0]PETSC ERROR: #3 EPSSetUp_KrylovSchur_Slice() line 467 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> [0]PETSC ERROR: #4 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>> [0]PETSC ERROR: #5 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c >>> Solution method: krylovschur >>> [0]PETSC ERROR: #6 EPSSliceGetEPS() line 306 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> [0]PETSC ERROR: #7 EPSSetUp_KrylovSchur_Slice() line 416 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>> [0]PETSC ERROR: #8 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>> [0]PETSC ERROR: #9 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c >> > From jroman at dsic.upv.es Fri Nov 19 07:11:37 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Fri, 19 Nov 2021 14:11:37 +0100 Subject: [petsc-users] PCSHELL does not support getting factor matrix In-Reply-To: <7A8AE4DA-5CAB-48BC-AAF5-B2F8D24715F1@gmail.com> References: <108CFB57-A068-4ADF-9E75-BF0876D77174@dsic.upv.es> <18BFAFD7-ECDE-4096-BC98-2343326449C8@gmail.com> <7A8AE4DA-5CAB-48BC-AAF5-B2F8D24715F1@gmail.com> Message-ID: <48B16BEB-E194-42B2-97B4-05D0DAA9AF74@dsic.upv.es> Yes, spectrum slicing needs computing the inertia, and also it uses the KSPSolve interface. Furthermore, the current implementation is single vector only, no blocks. The only part of SLEPc that uses PCApply directly are Davidson solvers. I could try to adapt them to use PCMatApply(). Thanks for the suggestion. Jose > El 19 nov 2021, a las 13:46, Stefano Zampini escribi?: > > Oh, I see the stack trace now. This requires to compute the inertia? > >> On Nov 19, 2021, at 3:45 PM, Stefano Zampini wrote: >> >> Jose >> >> Now that we have the PCMatApply interface, you could switch to use that inside SLEPc. I guess you are using MatSolve, right? >> If not, the alternative is to have a PCFactorGetMatrix with creates on the fly an object that behaves like it. The problem is that we do not have a matching restore, and the new object created will be leaked. >> What do you think? >> >> >>> On Nov 19, 2021, at 12:41 PM, Jose E. Roman wrote: >>> >>> It is trying to call PCFactorGetMatrix() on your PC, but this operation is not supported by PCSHELL and it is not possible to set it via PetscObjectComposeFunction(). PCSHELL uses the PCShellSet* interface, that is restricted to a limited number of operations. >>> >>> Jose >>> >>> >>>> El 19 nov 2021, a las 6:30, Sam Guo escribi?: >>>> >>>> Dear PETSc dev team, >>>> I am implementing SLEPc interval option using shell matrix as follows: >>>> >>>> EPSGetST(eps, &st); >>>> STSetType(st, STSINVERT); >>>> STGetKSP(st, &ksp); >>>> KSPSetOperators(ksp, A, A); >>>> KSPSetType(ksp, KSPPREONLY); >>>> KSPGetPC(ksp, &pc); >>>> KSPGetPC(ksp, &pc) >>>> MatSetOption(A, MAT_SPD, PETSC_TRUE); >>>> PCSetType(pc, PCSHELL); >>>> PCShellSetContext(pc, &appCtx); >>>> PCShellSetApply(pc, applyPreconditioner); >>>> PetscObjectComposeFunction((PetscObject)pc,"PCFactorGetZeroPivot_C",PCFactorGetZeroPivot_C); >>>> >>>> When I run it, I get the following error. Any idea what I did wrong? Thanks a lot for your help. >>>> >>>> [0]PETSC ERROR: No support for this operation for this object type >>>> [0]PETSC ERROR: PC type does not support getting factor matrix >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 >>>> [0]PETSC ERROR: Unknown Name on a arch-starccmplus_serial_real named pl2usbvu0037pc.net.plm.eds.com by cd4hhv Thu Nov 18 21:11:45 2021 >>>> Number of iterations of the method: 0 >>>> [0]PETSC ERROR: Configure options --with-x=0 --with-fc=0 --with-debugging=1 --with-blaslapack-dir=/u/cd4hhv/dev2/mkl/2017.2-cda-001/linux/lib/intel64/../.. --with-mpi=0 -CFLAGS=-g -CXXFLAGS=-g --with-clean=1 --force --with-scalar-type=real >>>> [0]PETSC ERROR: #1 PCFactorGetMatrix() line 1332 in ../../../petsc/src/ksp/pc/interface/precon.c >>>> [0]PETSC ERROR: #2 EPSSliceGetInertia() line 340 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> [0]PETSC ERROR: #3 EPSSetUp_KrylovSchur_Slice() line 467 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> [0]PETSC ERROR: #4 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>>> [0]PETSC ERROR: #5 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c >>>> Solution method: krylovschur >>>> [0]PETSC ERROR: #6 EPSSliceGetEPS() line 306 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> [0]PETSC ERROR: #7 EPSSetUp_KrylovSchur_Slice() line 416 in ../../../slepc/src/eps/impls/krylov/krylovschur/ks-slice.c >>>> [0]PETSC ERROR: #8 EPSSetUp_KrylovSchur() line 146 in ../../../slepc/src/eps/impls/krylov/krylovschur/krylovschur.c >>>> [0]PETSC ERROR: #9 EPSSetUp() line 173 in ../../../slepc/src/eps/interface/epssetup.c >>> >> > From bsmith at petsc.dev Fri Nov 19 07:20:39 2021 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 19 Nov 2021 08:20:39 -0500 Subject: [petsc-users] How to check NULL pointer in Fortran In-Reply-To: References: Message-ID: <5CCCE137-D42C-4F1D-8690-195A53504DED@petsc.dev> it is very possible there is a bug in the Fortran interface for this function. It looks like whoever wrote the Fortran interface did not think through the special case of contiguous entries. PETSC_EXTERN void petscsfgetgraph_(PetscSF *sf,PetscInt *nroots,PetscInt *nleaves, F90Array1d *ailocal, F90Array1d *airemote, PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(pilocal) PETSC_F90_2PTR_PROTO(piremote)) { const PetscInt *ilocal; const PetscSFNode *iremote; *ierr = PetscSFGetGraph(*sf,nroots,nleaves,&ilocal,&iremote);if (*ierr) return; *ierr = F90Array1dCreate((void*)ilocal,MPIU_INT,1,*nleaves, ailocal PETSC_F90_2PTR_PARAM(pilocal)); /* this creates a memory leak */ f90arraysfnodecreate_((PetscInt*)iremote,nleaves, airemote PETSC_F90_2PTR_PARAM(piremote)); } PetscErrorCode F90Array1dCreate(void *array,MPI_Datatype type,PetscInt start,PetscInt len,F90Array1d *ptr PETSC_F90_2PTR_PROTO(ptrd)) { PetscFunctionBegin; if (type == MPIU_SCALAR) { if (!len) array = PETSC_NULL_SCALAR_Fortran; f90array1dcreatescalar_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); } else if (type == MPIU_REAL) { if (!len) array = PETSC_NULL_REAL_Fortran; f90array1dcreatereal_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); } else if (type == MPIU_INT) { if (!len) array = PETSC_NULL_INTEGER_Fortran; f90array1dcreateint_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); } else if (type == MPIU_FORTRANADDR) { f90array1dcreatefortranaddr_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); } else SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Unsupported MPI_Datatype"); PetscFunctionReturn(0); } It blindly calls F90Array1dCreate() with a len = *nleaves > 0 but a NULL array ilocal then f90array1dcreateint_ must create an invalid object. Barry > On Nov 19, 2021, at 12:34 AM, ?? wrote: > > Dear PETSc-team, > > I am using function PetscSFGetGraph in my program like > ----------------------- > call PetscSFGetGraph(sf,gnroots,gnleaves,gmine,gremote,ierr) > ----------------------- > > In some cases, it works well. But in some cases, I encountered following error > ------------------ > PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range > ------------------ > I found it is due to the action of reading gmine, which is a fortran pointer point to an array. It is reasonable because PETSc manual tells me "if returned value is NULL, it means leaves are in contiguous storage". The problem is that I cannot find if gmine is a null pointer. > 1) I cannot use "if (gmine==PETSC_NULL_INTEGER)" because my intel compiler would return a compile error "A scalar-valued expression is required in this context" > 2) When using standard style of checking a null pointer in fortran, "associated(gmine)", it returns "T". Even in cases such action of "print *, gmine(1)" would give rise to above Segmentation Violation error. > Is there any means to check the NULL pointer in Fortran in above cases. > Many thanks, > Yuan From karthikeyan.chockalingam at stfc.ac.uk Fri Nov 19 09:46:59 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Fri, 19 Nov 2021 15:46:59 +0000 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> Message-ID: <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> Hello, I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view I profiled the run using NSYS - attached you find all the relevant files. Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active. I don?t see any cuda kernel relevant to solve. Is my assessment correct? How can I verify if hypre indeed is running on gpus? Best, Karthik. From: Mark Adams Date: Friday, 8 October 2021 at 18:47 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" Cc: "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] hypre on gpus I think you would want to use 'cuda' vec_type, but I . You might ask Hypre how one verifies that the GPU is used. Mark On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI > wrote: Yes, I used it for both cpu and gpu. Is that not okay? For gpu: -dm_mat_type hypre -dm_vec_type mpicuda For cpu: -dm_mat_type hypre -dm_vec_type mpi From: Mark Adams > Date: Friday, 8 October 2021 at 18:28 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Did you use -dm_mat_type hypre on the GPU case ? On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI > wrote: I tried a different exercise ran the same problem on two cpu cores and on two gpu: On gpu PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0 and on cpu PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00. I am not sure if that is any indication if hypre ran on gpus? Thanks, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 16:36 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI > wrote: The PCApply timing on gpu PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 and cpu PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 You don't have GPUs. probably. Use -dm_mat_type hypre. are close. It is hard for me tell if hypre on gpu is on or not. Best, Karthik. From: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Date: Friday, 8 October 2021 at 14:55 To: Mark Adams > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Thanks Mark, I will try your recommendations. Should I also change -dm_vec_type to hypre currently I have it as mpicuda? Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 14:33 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Hypre does not record its flops with PETSc's timers. Configure with and without CUDA and see if the timings change in PCApply. Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use -dm_mat_type hypre. Mark On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does not record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus. Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? Thanks, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ksp_cg_pc_hypre_ex45_N128_gpu_1_nsys.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: nsys_stat_hypre_128_1.txt URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screenshot 2021-11-18 at 17.21.49.png Type: image/png Size: 172194 bytes Desc: Screenshot 2021-11-18 at 17.21.49.png URL: From mfadams at lbl.gov Fri Nov 19 10:30:40 2021 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 19 Nov 2021 11:30:40 -0500 Subject: [petsc-users] hypre on gpus In-Reply-To: <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> Message-ID: You should run with -options_left to check that your options are being used. It may be -mat_type hypre. I have tested this: petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre* You can add -log_view and that will print performance data for each method like KSPSolve. If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU. In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre. I saw that the flops were 100% GPU in hypre. On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Hello, > > > > I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on > gpus > > > > ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre > -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg > -ksp_monitor -log_view > > > > I profiled the run using NSYS - attached you find all the relevant files. > > Looking at the profile I doubt if hypre is running on gpus. The cuda > kernels are barely active. > > I don?t see any cuda kernel relevant to solve. > > Is my assessment correct? How can I verify if hypre indeed is running on > gpus? > > > > Best, > > Karthik. > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:47 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > I think you would want to use 'cuda' vec_type, but I . > > You might ask Hypre how one verifies that the GPU is used. > > Mark > > > > On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Yes, I used it for both cpu and gpu. Is that not okay? > > > > For gpu: -dm_mat_type hypre -dm_vec_type mpicuda > > > > For cpu: -dm_mat_type hypre -dm_vec_type mpi > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:28 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Did you use -dm_mat_type hypre on the GPU case ? > > > > On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I tried a different exercise ran the same problem on two cpu cores and on > two gpu: > > > > On gpu > > > > PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 > 9.65e+01 0 > > > > and on cpu > > > > PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 > > > > timings again are close but gpu version did a reduction 6.0e+00 but the > cpu version did not 0.0e+00. > > I am not sure if that is any indication if hypre ran on gpus? > > > > Thanks, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 16:36 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > > > > > On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > The PCApply timing on > > > > gpu > > > > PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 > 0.00e+00 0 > > > > and cpu > > > > PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > > > > > > You don't have GPUs. probably. > > Use -dm_mat_type hypre. > > > > are close. It is hard for me tell if hypre on gpu is on or not. > > > > Best, > > Karthik. > > > > > > *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Date: *Friday, 8 October 2021 at 14:55 > *To: *Mark Adams > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Thanks Mark, I will try your recommendations. > > Should I also change -dm_vec_type to hypre currently I have it as mpicuda? > > > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 14:33 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Hypre does not record its flops with PETSc's timers. > > Configure with and without CUDA and see if the timings change in PCApply. > > Hypre does not dynamically switch between CUDA and CPU solves at > this time, but you want to use -dm_mat_type hypre. > > Mark > > > > On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have > attached the python configuration file and -log_view output from running > the below command options > > > > mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 > -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type > hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 > -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor > > > > The problem was solved and converged but from the output file I suspect > hypre is not running on gpus as PCApply and DMCreate does *not* record > any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are > running on gpus. > > > > Can you please let me know if I need to add any extra flag to the attached > arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? > > > > Thanks, > > Karthik. > > > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aduarteg at utexas.edu Fri Nov 19 11:39:29 2021 From: aduarteg at utexas.edu (Alfredo J Duarte Gomez) Date: Fri, 19 Nov 2021 11:39:29 -0600 Subject: [petsc-users] PC shell destroy In-Reply-To: <3F7565A5-E6BA-4C62-92ED-C5665BFC4B09@petsc.dev> References: <3F7565A5-E6BA-4C62-92ED-C5665BFC4B09@petsc.dev> Message-ID: Good morning, Sorry for picking this back up such a long time after the initial response, but I am now in a good place to implement this change after thoroughly testing the preconditioner shell alone. I have been looking for resources but it is still not clear to me how to use the MATSHELL for this purpose. Do I have to create my own MATSHELL that is basically just the solution and the shift itself, or at least operations that do this? Or is the matshell simply a dummy object that will allow me to interface the solution and ashift? In either of this cases I am still not sure how to begin this process, are there any helpful examples that I could model the MATSHELL from? KSP example ex15.c was incredibly helpful for the creation of my PCSHELL so it would be very helpful if there is something similar for the MATSHELL, and passing the ashift. Thank you, -Alfredo On Thu, Sep 30, 2021 at 5:09 PM Barry Smith wrote: > > Alfredo, > > I think the best approach for you to use is to have your own MATSHELL > and your own PCSHELL. You will use your MATSHELL as the second matrix > argument to TSSetIJacobian(). It should record the current x and the > current shift. > > Your PCSHELL will then, in PCSetUp(), get access to the current x and > the current shift from your MATSHELL and build itself. In other words most > of your > > / Get necessary objects from TS context > TSGetTime(ts,&time); > TSGetApplicationContext(ts,&user); > TSGetSolution(ts,&X); > TSGetTimeStep(ts,&dt); > TSGetStepNumber(ts, &stepi); > TSGetDM(ts,&da); > > tdt = time+dt; > // Update preconditioner context with current values > ierr = ShellPCSetUp(pc,da,tdt,dt,X,user);CHKERRQ(ierr); > > > code will disappear and you won't need to mess with the internals of the > TS (getting current dt etc) at all. What you need is handed off to your > TSSetIJacobian() function which will stick it into your MATSHELL. So nice > and clean code. > > Regarding the PCDestroy() for your PCSHELL. It only gets called when the > PC is finally destroyed which is when the TS is destroy. So if building > your PC in PCSetUp() requires creating new objects you should destroy any > previous ones when you create the new ones, hence "lost" objects won't > persist in the code. > > Barry > > > > > > On Sep 30, 2021, at 4:14 PM, Alfredo J Duarte Gomez > wrote: > > Good afternoon PETSC team, > > I am currently developing an application for PETSC in which I use my own > preconditioner with a PCSHELL. > > I have successfully set all the functions and the performance of the > preconditioner is good. I am using this PCSHELL within a TS object, and it > is imperative that the objects in the PCSHELL context are freed every time > since the memory requirements of those are large. > > I have set up the Preconditioner before the TS starts with the following > block of code: > > ---------------------------------------------------------------------------------------------------- > > ierr = PCSetType(pc,PCSHELL);CHKERRQ(ierr); > ierr = ShellPCCreate(&shell);CHKERRQ(ierr); > ierr = PCShellSetApply(pc,MatrixFreePreconditioner);CHKERRQ(ierr); > ierr = PCShellSetContext(pc,shell);CHKERRQ(ierr); > ierr = PCShellSetDestroy(pc,ShellPCDestroy);CHKERRQ(ierr); > ierr = PCShellSetName(pc,"MyPreconditioner");CHKERRQ(ierr); > ierr = ShellPCSetUp(pc,da,0.0,dt,u,user);CHKERRQ(ierr); > ierr = TSSetPreStep(ts,PreStep);CHKERRQ(ierr); > > > ------------------------------------------------------------------------------------------------ > > The shell context is then updated by using the following code within the > TSPreStep function: > > --------------------------------------------------------------------------- > > ierr = TSGetSNES(ts,&snes);CHKERRQ(ierr); > ierr = SNESGetKSP(snes,&ksp);CHKERRQ(ierr); > ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr); > > // Get necessary objects from TS context > TSGetTime(ts,&time); > TSGetApplicationContext(ts,&user); > TSGetSolution(ts,&X); > TSGetTimeStep(ts,&dt); > TSGetStepNumber(ts, &stepi); > TSGetDM(ts,&da); > > tdt = time+dt; > // Update preconditioner context with current values > ierr = ShellPCSetUp(pc,da,tdt,dt,X,user);CHKERRQ(ierr); > --------------------------------------------------------------------------- > > I have set up the necessary code in the function ShellPCDestroy to free > the objects within this context, however I am unsure when/if this function > is called automatically. Do I have to free the context myself after every > step? How would I call the function myself? > > I am running out of memory after a few steps, and I think this shell > context is the culprit. > > In addition to that, is it possible to get what is called the "ashift" dF/dU > + a*dF/dU_t in this function from the TS object? > > https://petsc.org/release/docs/manualpages/TS/TSSetIJacobian.html > > I need it as an input for my preconditioner (currrently hardcoded for > TSBEULER where ashift is always 1/dt). > > Thank you, > > -Alfredo > > -- > Alfredo Duarte > Graduate Research Assistant > The University of Texas at Austin > > > -- Alfredo Duarte Graduate Research Assistant The University of Texas at Austin -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.pelletier at protonmail.com Mon Nov 22 04:37:47 2021 From: milan.pelletier at protonmail.com (Milan Pelletier) Date: Mon, 22 Nov 2021 10:37:47 +0000 Subject: [petsc-users] Issues with log_view on windows Message-ID: Dear PETSc team, The way some options are passed to PETSc can be an issue on Windows, in particular regarding the "log_view" option. Since the colon ':' character is used as a separator, I do not see how absolute paths should be provided on Windows (since the drive name includes a colon). Is there a trick / workaround to overcome this? Besides, when I try to use the ascii_xml format, it crashes in file xmllogevent.c, on line 751: illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; (since nestedEvents is NULL). I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). Thanks for your help, Best regards, Milan Pelletier -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 22 05:37:59 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 22 Nov 2021 11:37:59 +0000 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> Message-ID: <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Thank you for your response. I tried to run the same example petsc/src/ksp/ksp/tutorials$ ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre but it crashed with the below error [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() Below are the options used to configure hypre with cuda support. Do you spot any mistakes? --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 Best, Karthik. From: Mark Adams Date: Friday, 19 November 2021 at 16:31 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" Cc: "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] hypre on gpus You should run with -options_left to check that your options are being used. It may be -mat_type hypre. I have tested this: petsc/src/ksp/ksp/tutorials$ srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre You can add -log_view and that will print performance data for each method like KSPSolve. If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU. In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre. I saw that the flops were 100% GPU in hypre. On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view I profiled the run using NSYS - attached you find all the relevant files. Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active. I don?t see any cuda kernel relevant to solve. Is my assessment correct? How can I verify if hypre indeed is running on gpus? Best, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 18:47 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus I think you would want to use 'cuda' vec_type, but I . You might ask Hypre how one verifies that the GPU is used. Mark On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI > wrote: Yes, I used it for both cpu and gpu. Is that not okay? For gpu: -dm_mat_type hypre -dm_vec_type mpicuda For cpu: -dm_mat_type hypre -dm_vec_type mpi From: Mark Adams > Date: Friday, 8 October 2021 at 18:28 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Did you use -dm_mat_type hypre on the GPU case ? On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI > wrote: I tried a different exercise ran the same problem on two cpu cores and on two gpu: On gpu PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0 and on cpu PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00. I am not sure if that is any indication if hypre ran on gpus? Thanks, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 16:36 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI > wrote: The PCApply timing on gpu PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 and cpu PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 You don't have GPUs. probably. Use -dm_mat_type hypre. are close. It is hard for me tell if hypre on gpu is on or not. Best, Karthik. From: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Date: Friday, 8 October 2021 at 14:55 To: Mark Adams > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Thanks Mark, I will try your recommendations. Should I also change -dm_vec_type to hypre currently I have it as mpicuda? Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 14:33 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Hypre does not record its flops with PETSc's timers. Configure with and without CUDA and see if the timings change in PCApply. Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use -dm_mat_type hypre. Mark On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does not record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus. Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? Thanks, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 22 07:39:44 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 08:39:44 -0500 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < petsc-users at mcs.anl.gov> wrote: > Dear PETSc team, > > The way some options are passed to PETSc can be an issue on Windows, in > particular regarding the "log_view" option. > Since the colon ':' character is used as a separator, I do not see how > absolute paths should be provided on Windows (since the drive name includes > a colon). Is there a trick / workaround to overcome this? > I thought we used Cygwin paths, but maybe one of the Windows people knows better. > Besides, when I try to use the ascii_xml format, it crashes in file > xmllogevent.c, on line 751: > illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > (since nestedEvents is NULL). > This likely means that you have unbalanced events. I can run this on examples. For instance does SNES ex5 work for you? Thanks, Matt > I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > > Thanks for your help, > Best regards, > Milan Pelletier > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 22 07:41:19 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 08:41:19 -0500 Subject: [petsc-users] hypre on gpus In-Reply-To: <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Thank you for your response. I tried to run the same example > > > > petsc/src/ksp/ksp/tutorials$ *./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > but it crashed with the below error > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > Hi Karthik, Please do not clip the error message. We need the entire output. This seems strange that you would get a logic error, since that should be the same on any architecture. So could you also send the make.log? Thanks, Matt > Below are the options used to configure hypre with cuda support. Do you > spot any mistakes? > > > > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > > > --with-cuda=1 > > --with-cuda-arch=70 > > --download-hypre=yes > > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > > --download-hypre-commit=origin/hypre_petsc > > --with-shared-libraries=1 > > --known-mpi-shared-libraries=1 > > --with-cc=mpicc > > --with-cxx=mpicxx > > -with-fc=mpif90 > > > > Best, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 19 November 2021 at 16:31 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > You should run with -options_left to check that your options are being > used. It may be -mat_type hypre. > > > > I have tested this: > > > > petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > You can add -log_view and that will print performance data for each method > like KSPSolve. > > > > If you are configured with a GPU there will be some extra columns that > give the percent of flops on the GPU. > > > > In the past hypre has not registered flops with us, but I do get flops > from hypre now, and -ksp_view showed that it did indeed use hypre. > > I saw that the flops were 100% GPU in hypre. > > > > > > On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on > gpus > > > > ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre > -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg > -ksp_monitor -log_view > > > > I profiled the run using NSYS - attached you find all the relevant files. > > Looking at the profile I doubt if hypre is running on gpus. The cuda > kernels are barely active. > > I don?t see any cuda kernel relevant to solve. > > Is my assessment correct? How can I verify if hypre indeed is running on > gpus? > > > > Best, > > Karthik. > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:47 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > I think you would want to use 'cuda' vec_type, but I . > > You might ask Hypre how one verifies that the GPU is used. > > Mark > > > > On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Yes, I used it for both cpu and gpu. Is that not okay? > > > > For gpu: -dm_mat_type hypre -dm_vec_type mpicuda > > > > For cpu: -dm_mat_type hypre -dm_vec_type mpi > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:28 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Did you use -dm_mat_type hypre on the GPU case ? > > > > On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I tried a different exercise ran the same problem on two cpu cores and on > two gpu: > > > > On gpu > > > > PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 > 9.65e+01 0 > > > > and on cpu > > > > PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 > > > > timings again are close but gpu version did a reduction 6.0e+00 but the > cpu version did not 0.0e+00. > > I am not sure if that is any indication if hypre ran on gpus? > > > > Thanks, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 16:36 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > > > > > On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > The PCApply timing on > > > > gpu > > > > PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 > 0.00e+00 0 > > > > and cpu > > > > PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > > > > > > You don't have GPUs. probably. > > Use -dm_mat_type hypre. > > > > are close. It is hard for me tell if hypre on gpu is on or not. > > > > Best, > > Karthik. > > > > > > *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Date: *Friday, 8 October 2021 at 14:55 > *To: *Mark Adams > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Thanks Mark, I will try your recommendations. > > Should I also change -dm_vec_type to hypre currently I have it as mpicuda? > > > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 14:33 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Hypre does not record its flops with PETSc's timers. > > Configure with and without CUDA and see if the timings change in PCApply. > > Hypre does not dynamically switch between CUDA and CPU solves at > this time, but you want to use -dm_mat_type hypre. > > Mark > > > > On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have > attached the python configuration file and -log_view output from running > the below command options > > > > mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 > -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type > hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 > -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor > > > > The problem was solved and converged but from the output file I suspect > hypre is not running on gpus as PCApply and DMCreate does *not* record > any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are > running on gpus. > > > > Can you please let me know if I need to add any extra flag to the attached > arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? > > > > Thanks, > > Karthik. > > > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 22 07:50:41 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 22 Nov 2021 13:50:41 +0000 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: Hi Matt, Below is the entire error message: [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 11:33:41 2021 [0]PETSC ERROR: Configure options --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 --with-debugging=yes --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 [0]PETSC ERROR: #1 MatGetOwnershipRange() at /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 [0]PETSC ERROR: #2 main() at ex4.c:40 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -ksp_view [0]PETSC ERROR: -mat_type hypre [0]PETSC ERROR: -pc_type hypre [0]PETSC ERROR: -use_gpu_aware_mpi 0 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- I have also attached the make.log. Thank you for having a look. Best, Karthik. From: Matthew Knepley Date: Monday, 22 November 2021 at 13:41 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" Cc: Mark Adams , "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] hypre on gpus On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thank you for your response. I tried to run the same example petsc/src/ksp/ksp/tutorials$ ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre but it crashed with the below error [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() Hi Karthik, Please do not clip the error message. We need the entire output. This seems strange that you would get a logic error, since that should be the same on any architecture. So could you also send the make.log? Thanks, Matt Below are the options used to configure hypre with cuda support. Do you spot any mistakes? --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 Best, Karthik. From: Mark Adams > Date: Friday, 19 November 2021 at 16:31 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus You should run with -options_left to check that your options are being used. It may be -mat_type hypre. I have tested this: petsc/src/ksp/ksp/tutorials$ srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre You can add -log_view and that will print performance data for each method like KSPSolve. If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU. In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre. I saw that the flops were 100% GPU in hypre. On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view I profiled the run using NSYS - attached you find all the relevant files. Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active. I don?t see any cuda kernel relevant to solve. Is my assessment correct? How can I verify if hypre indeed is running on gpus? Best, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 18:47 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus I think you would want to use 'cuda' vec_type, but I . You might ask Hypre how one verifies that the GPU is used. Mark On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI > wrote: Yes, I used it for both cpu and gpu. Is that not okay? For gpu: -dm_mat_type hypre -dm_vec_type mpicuda For cpu: -dm_mat_type hypre -dm_vec_type mpi From: Mark Adams > Date: Friday, 8 October 2021 at 18:28 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Did you use -dm_mat_type hypre on the GPU case ? On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI > wrote: I tried a different exercise ran the same problem on two cpu cores and on two gpu: On gpu PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0 and on cpu PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00. I am not sure if that is any indication if hypre ran on gpus? Thanks, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 16:36 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI > wrote: The PCApply timing on gpu PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 and cpu PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 You don't have GPUs. probably. Use -dm_mat_type hypre. are close. It is hard for me tell if hypre on gpu is on or not. Best, Karthik. From: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Date: Friday, 8 October 2021 at 14:55 To: Mark Adams > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Thanks Mark, I will try your recommendations. Should I also change -dm_vec_type to hypre currently I have it as mpicuda? Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 14:33 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Hypre does not record its flops with PETSc's timers. Configure with and without CUDA and see if the timings change in PCApply. Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use -dm_mat_type hypre. Mark On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does not record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus. Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? Thanks, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: make.log Type: application/octet-stream Size: 118056 bytes Desc: make.log URL: From knepley at gmail.com Mon Nov 22 08:28:54 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 09:28:54 -0500 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Hi Matt, > > > > Below is the entire error message: > I cannot reproduce this: main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre KSP Object: 1 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 1 MPI processes type: hypre rows=56, cols=56 Norm of error 8.69801e-05 iterations 2 This is on the 'main' branch. So either there is some bug in release, or something is strange on your end. Since we run Hypre tests for the CI, I am leaning toward the latter. Can you try the 'main' branch? We will have to use this anyway if we want any fixes. Thanks, Matt > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 > > [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 > 11:33:41 2021 > > [0]PETSC ERROR: Configure options > --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 > --with-debugging=yes > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 > --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx > -with-fc=mpif90 > > [0]PETSC ERROR: #1 MatGetOwnershipRange() at > /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 > > [0]PETSC ERROR: #2 main() at ex4.c:40 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -ksp_type cg > > [0]PETSC ERROR: -ksp_view > > [0]PETSC ERROR: -mat_type hypre > > [0]PETSC ERROR: -pc_type hypre > > [0]PETSC ERROR: -use_gpu_aware_mpi 0 > > *[0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov----------* > > -------------------------------------------------------------------------- > > > > I have also attached the make.log. Thank you for having a look. > > > > Best, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Monday, 22 November 2021 at 13:41 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Mark Adams , "petsc-users at mcs.anl.gov" < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] hypre on gpus > > > > On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thank you for your response. I tried to run the same example > > > > petsc/src/ksp/ksp/tutorials$ *./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > but it crashed with the below error > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > > > Hi Karthik, > > > > Please do not clip the error message. We need the entire output. This > seems strange that you would get a logic error, > > since that should be the same on any architecture. So could you also send > the make.log? > > > > Thanks, > > > > Matt > > > > Below are the options used to configure hypre with cuda support. Do you > spot any mistakes? > > > > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > > > --with-cuda=1 > > --with-cuda-arch=70 > > --download-hypre=yes > > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > > --download-hypre-commit=origin/hypre_petsc > > --with-shared-libraries=1 > > --known-mpi-shared-libraries=1 > > --with-cc=mpicc > > --with-cxx=mpicxx > > -with-fc=mpif90 > > > > Best, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 19 November 2021 at 16:31 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > You should run with -options_left to check that your options are being > used. It may be -mat_type hypre. > > > > I have tested this: > > > > petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > You can add -log_view and that will print performance data for each method > like KSPSolve. > > > > If you are configured with a GPU there will be some extra columns that > give the percent of flops on the GPU. > > > > In the past hypre has not registered flops with us, but I do get flops > from hypre now, and -ksp_view showed that it did indeed use hypre. > > I saw that the flops were 100% GPU in hypre. > > > > > > On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on > gpus > > > > ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre > -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg > -ksp_monitor -log_view > > > > I profiled the run using NSYS - attached you find all the relevant files. > > Looking at the profile I doubt if hypre is running on gpus. The cuda > kernels are barely active. > > I don?t see any cuda kernel relevant to solve. > > Is my assessment correct? How can I verify if hypre indeed is running on > gpus? > > > > Best, > > Karthik. > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:47 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > I think you would want to use 'cuda' vec_type, but I . > > You might ask Hypre how one verifies that the GPU is used. > > Mark > > > > On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Yes, I used it for both cpu and gpu. Is that not okay? > > > > For gpu: -dm_mat_type hypre -dm_vec_type mpicuda > > > > For cpu: -dm_mat_type hypre -dm_vec_type mpi > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:28 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Did you use -dm_mat_type hypre on the GPU case ? > > > > On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I tried a different exercise ran the same problem on two cpu cores and on > two gpu: > > > > On gpu > > > > PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 > 9.65e+01 0 > > > > and on cpu > > > > PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 > > > > timings again are close but gpu version did a reduction 6.0e+00 but the > cpu version did not 0.0e+00. > > I am not sure if that is any indication if hypre ran on gpus? > > > > Thanks, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 16:36 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > > > > > On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > The PCApply timing on > > > > gpu > > > > PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 > 0.00e+00 0 > > > > and cpu > > > > PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > > > > > > You don't have GPUs. probably. > > Use -dm_mat_type hypre. > > > > are close. It is hard for me tell if hypre on gpu is on or not. > > > > Best, > > Karthik. > > > > > > *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Date: *Friday, 8 October 2021 at 14:55 > *To: *Mark Adams > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Thanks Mark, I will try your recommendations. > > Should I also change -dm_vec_type to hypre currently I have it as mpicuda? > > > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 14:33 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Hypre does not record its flops with PETSc's timers. > > Configure with and without CUDA and see if the timings change in PCApply. > > Hypre does not dynamically switch between CUDA and CPU solves at > this time, but you want to use -dm_mat_type hypre. > > Mark > > > > On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have > attached the python configuration file and -log_view output from running > the below command options > > > > mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 > -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type > hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 > -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor > > > > The problem was solved and converged but from the output file I suspect > hypre is not running on gpus as PCApply and DMCreate does *not* record > any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are > running on gpus. > > > > Can you please let me know if I need to add any extra flag to the attached > arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? > > > > Thanks, > > Karthik. > > > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefano.zampini at gmail.com Mon Nov 22 08:45:56 2021 From: stefano.zampini at gmail.com (Stefano Zampini) Date: Mon, 22 Nov 2021 17:45:56 +0300 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: You don't need to specify the HYPRE commit. Remove --download-hypre-commit=origin/hypre_petsc from the configuration options Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley ha scritto: > On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > >> Hi Matt, >> >> >> >> Below is the entire error message: >> > > I cannot reproduce this: > > main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view > -ksp_type cg -mat_type hypre -pc_type hypre > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: hypre > rows=56, cols=56 > Norm of error 8.69801e-05 iterations 2 > > This is on the 'main' branch. So either there is some bug in release, or > something is strange on your end. Since we run Hypre tests for the CI, > I am leaning toward the latter. Can you try the 'main' branch? We will > have to use this anyway if we want any fixes. > > Thanks, > > Matt > > >> *[0]PETSC ERROR: --------------------- Error Message >> --------------------------------------------------------------* >> >> [0]PETSC ERROR: Object is in wrong state >> >> [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the >> matrix has not yet been factored on argument 1 "mat" before >> MatGetOwnershipRange() >> >> [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. >> >> [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 >> >> [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 >> 11:33:41 2021 >> >> [0]PETSC ERROR: Configure options >> --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 >> --with-debugging=yes >> --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl >> --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes >> --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 >> --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 >> --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx >> -with-fc=mpif90 >> >> [0]PETSC ERROR: #1 MatGetOwnershipRange() at >> /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 >> >> [0]PETSC ERROR: #2 main() at ex4.c:40 >> >> [0]PETSC ERROR: PETSc Option Table entries: >> >> [0]PETSC ERROR: -ksp_type cg >> >> [0]PETSC ERROR: -ksp_view >> >> [0]PETSC ERROR: -mat_type hypre >> >> [0]PETSC ERROR: -pc_type hypre >> >> [0]PETSC ERROR: -use_gpu_aware_mpi 0 >> >> *[0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov----------* >> >> -------------------------------------------------------------------------- >> >> >> >> I have also attached the make.log. Thank you for having a look. >> >> >> >> Best, >> >> Karthik. >> >> >> >> *From: *Matthew Knepley >> *Date: *Monday, 22 November 2021 at 13:41 >> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Cc: *Mark Adams , "petsc-users at mcs.anl.gov" < >> petsc-users at mcs.anl.gov> >> *Subject: *Re: [petsc-users] hypre on gpus >> >> >> >> On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI < >> karthikeyan.chockalingam at stfc.ac.uk> wrote: >> >> Thank you for your response. I tried to run the same example >> >> >> >> petsc/src/ksp/ksp/tutorials$ *./ex4 -ksp_type cg -mat_type hypre >> -ksp_view -pc_type hypre* >> >> >> >> but it crashed with the below error >> >> >> >> *[0]PETSC ERROR: --------------------- Error Message >> --------------------------------------------------------------* >> >> [0]PETSC ERROR: Object is in wrong state >> >> [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the >> matrix has not yet been factored on argument 1 "mat" before >> MatGetOwnershipRange() >> >> >> >> Hi Karthik, >> >> >> >> Please do not clip the error message. We need the entire output. This >> seems strange that you would get a logic error, >> >> since that should be the same on any architecture. So could you also send >> the make.log? >> >> >> >> Thanks, >> >> >> >> Matt >> >> >> >> Below are the options used to configure hypre with cuda support. Do you >> spot any mistakes? >> >> >> >> --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl >> >> >> --with-cuda=1 >> >> --with-cuda-arch=70 >> >> --download-hypre=yes >> >> --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 >> >> --download-hypre-commit=origin/hypre_petsc >> >> --with-shared-libraries=1 >> >> --known-mpi-shared-libraries=1 >> >> --with-cc=mpicc >> >> --with-cxx=mpicxx >> >> -with-fc=mpif90 >> >> >> >> Best, >> >> Karthik. >> >> >> >> >> >> *From: *Mark Adams >> *Date: *Friday, 19 November 2021 at 16:31 >> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Cc: *"petsc-users at mcs.anl.gov" >> *Subject: *Re: [petsc-users] hypre on gpus >> >> >> >> You should run with -options_left to check that your options are being >> used. It may be -mat_type hypre. >> >> >> >> I have tested this: >> >> >> >> petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type >> hypre -ksp_view -pc_type hypre* >> >> >> >> You can add -log_view and that will print performance data for each >> method like KSPSolve. >> >> >> >> If you are configured with a GPU there will be some extra columns that >> give the percent of flops on the GPU. >> >> >> >> In the past hypre has not registered flops with us, but I do get flops >> from hypre now, and -ksp_view showed that it did indeed use hypre. >> >> I saw that the flops were 100% GPU in hypre. >> >> >> >> >> >> On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI < >> karthikeyan.chockalingam at stfc.ac.uk> wrote: >> >> Hello, >> >> >> >> I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on >> gpus >> >> >> >> ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre >> -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg >> -ksp_monitor -log_view >> >> >> >> I profiled the run using NSYS - attached you find all the relevant >> files. >> >> Looking at the profile I doubt if hypre is running on gpus. The cuda >> kernels are barely active. >> >> I don?t see any cuda kernel relevant to solve. >> >> Is my assessment correct? How can I verify if hypre indeed is running on >> gpus? >> >> >> >> Best, >> >> Karthik. >> >> >> >> *From: *Mark Adams >> *Date: *Friday, 8 October 2021 at 18:47 >> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Cc: *"petsc-users at mcs.anl.gov" >> *Subject: *Re: [petsc-users] hypre on gpus >> >> >> >> I think you would want to use 'cuda' vec_type, but I . >> >> You might ask Hypre how one verifies that the GPU is used. >> >> Mark >> >> >> >> On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI < >> karthikeyan.chockalingam at stfc.ac.uk> wrote: >> >> Yes, I used it for both cpu and gpu. Is that not okay? >> >> >> >> For gpu: -dm_mat_type hypre -dm_vec_type mpicuda >> >> >> >> For cpu: -dm_mat_type hypre -dm_vec_type mpi >> >> >> >> *From: *Mark Adams >> *Date: *Friday, 8 October 2021 at 18:28 >> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Cc: *"petsc-users at mcs.anl.gov" >> *Subject: *Re: [petsc-users] hypre on gpus >> >> >> >> Did you use -dm_mat_type hypre on the GPU case ? >> >> >> >> On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < >> karthikeyan.chockalingam at stfc.ac.uk> wrote: >> >> I tried a different exercise ran the same problem on two cpu cores and on >> two gpu: >> >> >> >> On gpu >> >> >> >> PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 >> 9.65e+01 0 >> >> >> >> and on cpu >> >> >> >> PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 >> >> >> >> timings again are close but gpu version did a reduction 6.0e+00 but the >> cpu version did not 0.0e+00. >> >> I am not sure if that is any indication if hypre ran on gpus? >> >> >> >> Thanks, >> >> Karthik. >> >> >> >> >> >> *From: *Mark Adams >> *Date: *Friday, 8 October 2021 at 16:36 >> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Cc: *"petsc-users at mcs.anl.gov" >> *Subject: *Re: [petsc-users] hypre on gpus >> >> >> >> >> >> >> >> On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < >> karthikeyan.chockalingam at stfc.ac.uk> wrote: >> >> The PCApply timing on >> >> >> >> gpu >> >> >> >> PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 >> 0.00e+00 0 >> >> >> >> and cpu >> >> >> >> PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 >> 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 >> >> >> >> >> >> You don't have GPUs. probably. >> >> Use -dm_mat_type hypre. >> >> >> >> are close. It is hard for me tell if hypre on gpu is on or not. >> >> >> >> Best, >> >> Karthik. >> >> >> >> >> >> *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Date: *Friday, 8 October 2021 at 14:55 >> *To: *Mark Adams >> *Cc: *"petsc-users at mcs.anl.gov" >> *Subject: *Re: [petsc-users] hypre on gpus >> >> >> >> Thanks Mark, I will try your recommendations. >> >> Should I also change -dm_vec_type to hypre currently I have it as >> mpicuda? >> >> >> >> Karthik. >> >> >> >> >> >> *From: *Mark Adams >> *Date: *Friday, 8 October 2021 at 14:33 >> *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < >> karthikeyan.chockalingam at stfc.ac.uk> >> *Cc: *"petsc-users at mcs.anl.gov" >> *Subject: *Re: [petsc-users] hypre on gpus >> >> >> >> Hypre does not record its flops with PETSc's timers. >> >> Configure with and without CUDA and see if the timings change in PCApply. >> >> Hypre does not dynamically switch between CUDA and CPU solves at >> this time, but you want to use -dm_mat_type hypre. >> >> Mark >> >> >> >> On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < >> karthikeyan.chockalingam at stfc.ac.uk> wrote: >> >> Hello, >> >> >> >> I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have >> attached the python configuration file and -log_view output from running >> the below command options >> >> >> >> mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z >> 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres >> -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 >> -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor >> >> >> >> The problem was solved and converged but from the output file I suspect >> hypre is not running on gpus as PCApply and DMCreate does *not* record >> any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are >> running on gpus. >> >> >> >> Can you please let me know if I need to add any extra flag to the >> attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working >> on gpus? >> >> >> >> Thanks, >> >> Karthik. >> >> >> >> >> >> This email and any attachments are intended solely for the use of the >> named recipients. If you are not the intended recipient you must not use, >> disclose, copy or distribute this email or any of its attachments and >> should notify the sender immediately and delete this email from your >> system. UK Research and Innovation (UKRI) has taken every reasonable >> precaution to minimise risk of this email or any attachments containing >> viruses or malware but the recipient should carry out its own virus and >> malware checks before opening the attachments. UKRI does not accept any >> liability for any losses or damages which the recipient may sustain due to >> presence of any viruses. >> >> >> >> >> -- >> >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 22 09:20:15 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 22 Nov 2021 15:20:15 +0000 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: Thanks Stefano, I have another build without the --download-hypre-commit=origin/hypre_petsc but that gives a different error. [kchockalingam at glados tutorials]$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HYPRE_IJMatrixAssemble(): error code 12 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.15.3, Aug 06, 2021 [0]PETSC ERROR: ./ex4 on a named glados.dl.ac.uk by kchockalingam Mon Nov 22 15:07:46 2021 [0]PETSC ERROR: Configure options --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer" FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1 --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --with-debugging=no PETSC_ARCH=arch-ci-linux-cuda11-double [0]PETSC ERROR: #1 MatAssemblyEnd_HYPRE() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/impls/hypre/mhypre.c:1212 [0]PETSC ERROR: #2 MatAssemblyEnd() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/interface/matrix.c:5652 [0]PETSC ERROR: #3 main() at ex4.c:84 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -ksp_view [0]PETSC ERROR: -mat_type hypre [0]PETSC ERROR: -pc_type hypre [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- Best, Karthik. From: Stefano Zampini Date: Monday, 22 November 2021 at 14:46 To: Matthew Knepley Cc: "Chockalingam, Karthikeyan (STFC,DL,HC)" , "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] hypre on gpus You don't need to specify the HYPRE commit. Remove --download-hypre-commit=origin/hypre_petsc from the configuration options Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley > ha scritto: On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, Below is the entire error message: I cannot reproduce this: main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre KSP Object: 1 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 1 MPI processes type: hypre rows=56, cols=56 Norm of error 8.69801e-05 iterations 2 This is on the 'main' branch. So either there is some bug in release, or something is strange on your end. Since we run Hypre tests for the CI, I am leaning toward the latter. Can you try the 'main' branch? We will have to use this anyway if we want any fixes. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 11:33:41 2021 [0]PETSC ERROR: Configure options --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 --with-debugging=yes --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 [0]PETSC ERROR: #1 MatGetOwnershipRange() at /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 [0]PETSC ERROR: #2 main() at ex4.c:40 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -ksp_view [0]PETSC ERROR: -mat_type hypre [0]PETSC ERROR: -pc_type hypre [0]PETSC ERROR: -use_gpu_aware_mpi 0 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- I have also attached the make.log. Thank you for having a look. Best, Karthik. From: Matthew Knepley > Date: Monday, 22 November 2021 at 13:41 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: Mark Adams >, "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thank you for your response. I tried to run the same example petsc/src/ksp/ksp/tutorials$ ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre but it crashed with the below error [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() Hi Karthik, Please do not clip the error message. We need the entire output. This seems strange that you would get a logic error, since that should be the same on any architecture. So could you also send the make.log? Thanks, Matt Below are the options used to configure hypre with cuda support. Do you spot any mistakes? --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 Best, Karthik. From: Mark Adams > Date: Friday, 19 November 2021 at 16:31 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus You should run with -options_left to check that your options are being used. It may be -mat_type hypre. I have tested this: petsc/src/ksp/ksp/tutorials$ srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre You can add -log_view and that will print performance data for each method like KSPSolve. If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU. In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre. I saw that the flops were 100% GPU in hypre. On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view I profiled the run using NSYS - attached you find all the relevant files. Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active. I don?t see any cuda kernel relevant to solve. Is my assessment correct? How can I verify if hypre indeed is running on gpus? Best, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 18:47 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus I think you would want to use 'cuda' vec_type, but I . You might ask Hypre how one verifies that the GPU is used. Mark On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI > wrote: Yes, I used it for both cpu and gpu. Is that not okay? For gpu: -dm_mat_type hypre -dm_vec_type mpicuda For cpu: -dm_mat_type hypre -dm_vec_type mpi From: Mark Adams > Date: Friday, 8 October 2021 at 18:28 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Did you use -dm_mat_type hypre on the GPU case ? On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI > wrote: I tried a different exercise ran the same problem on two cpu cores and on two gpu: On gpu PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0 and on cpu PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00. I am not sure if that is any indication if hypre ran on gpus? Thanks, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 16:36 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI > wrote: The PCApply timing on gpu PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 and cpu PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 You don't have GPUs. probably. Use -dm_mat_type hypre. are close. It is hard for me tell if hypre on gpu is on or not. Best, Karthik. From: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Date: Friday, 8 October 2021 at 14:55 To: Mark Adams > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Thanks Mark, I will try your recommendations. Should I also change -dm_vec_type to hypre currently I have it as mpicuda? Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 14:33 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Hypre does not record its flops with PETSc's timers. Configure with and without CUDA and see if the timings change in PCApply. Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use -dm_mat_type hypre. Mark On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does not record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus. Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? Thanks, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- Stefano -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 22 09:25:39 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 10:25:39 -0500 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: On Mon, Nov 22, 2021 at 10:20 AM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Thanks Stefano, > I just did a clean build with no problems. Can you try a clean build of main? Thanks, Matt > > > I have another build without the --download-hypre-commit=origin/hypre_petsc > but that gives a different error. > > > > [kchockalingam at glados tutorials]$ ./ex4 -ksp_view -ksp_type cg -mat_type > hypre -pc_type hypre > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Error in external library > > [0]PETSC ERROR: Error in HYPRE_IJMatrixAssemble(): error code 12 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.15.3, Aug 06, 2021 > > [0]PETSC ERROR: ./ex4 on a named glados.dl.ac.uk by kchockalingam Mon > Nov 22 15:07:46 2021 > > [0]PETSC ERROR: Configure options > --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs > --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer" > FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 > -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 > --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ > --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1 > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --with-debugging=no > PETSC_ARCH=arch-ci-linux-cuda11-double > > [0]PETSC ERROR: #1 MatAssemblyEnd_HYPRE() at > /home/kchockalingam/tools/petsc-3.15.3/src/mat/impls/hypre/mhypre.c:1212 > > [0]PETSC ERROR: #2 MatAssemblyEnd() at > /home/kchockalingam/tools/petsc-3.15.3/src/mat/interface/matrix.c:5652 > > [0]PETSC ERROR: #3 main() at ex4.c:84 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -ksp_type cg > > [0]PETSC ERROR: -ksp_view > > [0]PETSC ERROR: -mat_type hypre > > [0]PETSC ERROR: -pc_type hypre > > *[0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov----------* > > -------------------------------------------------------------------------- > > > > Best, > > Karthik. > > > > *From: *Stefano Zampini > *Date: *Monday, 22 November 2021 at 14:46 > *To: *Matthew Knepley > *Cc: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk>, "petsc-users at mcs.anl.gov" < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] hypre on gpus > > > > You don't need to specify the HYPRE commit. Remove --download-hypre-commit=origin/hypre_petsc > from the configuration options > > > > Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley < > knepley at gmail.com> ha scritto: > > On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > Below is the entire error message: > > > > I cannot reproduce this: > > > > main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view > -ksp_type cg -mat_type hypre -pc_type hypre > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: hypre > rows=56, cols=56 > Norm of error 8.69801e-05 iterations 2 > > This is on the 'main' branch. So either there is some bug in release, or > something is strange on your end. Since we run Hypre tests for the CI, > > I am leaning toward the latter. Can you try the 'main' branch? We will > have to use this anyway if we want any fixes. > > > > Thanks, > > > > Matt > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 > > [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 > 11:33:41 2021 > > [0]PETSC ERROR: Configure options > --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 > --with-debugging=yes > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 > --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx > -with-fc=mpif90 > > [0]PETSC ERROR: #1 MatGetOwnershipRange() at > /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 > > [0]PETSC ERROR: #2 main() at ex4.c:40 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -ksp_type cg > > [0]PETSC ERROR: -ksp_view > > [0]PETSC ERROR: -mat_type hypre > > [0]PETSC ERROR: -pc_type hypre > > [0]PETSC ERROR: -use_gpu_aware_mpi 0 > > *[0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov----------* > > -------------------------------------------------------------------------- > > > > I have also attached the make.log. Thank you for having a look. > > > > Best, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Monday, 22 November 2021 at 13:41 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Mark Adams , "petsc-users at mcs.anl.gov" < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] hypre on gpus > > > > On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thank you for your response. I tried to run the same example > > > > petsc/src/ksp/ksp/tutorials$ *./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > but it crashed with the below error > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > > > Hi Karthik, > > > > Please do not clip the error message. We need the entire output. This > seems strange that you would get a logic error, > > since that should be the same on any architecture. So could you also send > the make.log? > > > > Thanks, > > > > Matt > > > > Below are the options used to configure hypre with cuda support. Do you > spot any mistakes? > > > > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > > > --with-cuda=1 > > --with-cuda-arch=70 > > --download-hypre=yes > > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > > --download-hypre-commit=origin/hypre_petsc > > --with-shared-libraries=1 > > --known-mpi-shared-libraries=1 > > --with-cc=mpicc > > --with-cxx=mpicxx > > -with-fc=mpif90 > > > > Best, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 19 November 2021 at 16:31 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > You should run with -options_left to check that your options are being > used. It may be -mat_type hypre. > > > > I have tested this: > > > > petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > You can add -log_view and that will print performance data for each method > like KSPSolve. > > > > If you are configured with a GPU there will be some extra columns that > give the percent of flops on the GPU. > > > > In the past hypre has not registered flops with us, but I do get flops > from hypre now, and -ksp_view showed that it did indeed use hypre. > > I saw that the flops were 100% GPU in hypre. > > > > > > On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on > gpus > > > > ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre > -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg > -ksp_monitor -log_view > > > > I profiled the run using NSYS - attached you find all the relevant files. > > Looking at the profile I doubt if hypre is running on gpus. The cuda > kernels are barely active. > > I don?t see any cuda kernel relevant to solve. > > Is my assessment correct? How can I verify if hypre indeed is running on > gpus? > > > > Best, > > Karthik. > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:47 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > I think you would want to use 'cuda' vec_type, but I . > > You might ask Hypre how one verifies that the GPU is used. > > Mark > > > > On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Yes, I used it for both cpu and gpu. Is that not okay? > > > > For gpu: -dm_mat_type hypre -dm_vec_type mpicuda > > > > For cpu: -dm_mat_type hypre -dm_vec_type mpi > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:28 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Did you use -dm_mat_type hypre on the GPU case ? > > > > On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I tried a different exercise ran the same problem on two cpu cores and on > two gpu: > > > > On gpu > > > > PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 > 9.65e+01 0 > > > > and on cpu > > > > PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 > > > > timings again are close but gpu version did a reduction 6.0e+00 but the > cpu version did not 0.0e+00. > > I am not sure if that is any indication if hypre ran on gpus? > > > > Thanks, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 16:36 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > > > > > On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > The PCApply timing on > > > > gpu > > > > PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 > 0.00e+00 0 > > > > and cpu > > > > PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > > > > > > You don't have GPUs. probably. > > Use -dm_mat_type hypre. > > > > are close. It is hard for me tell if hypre on gpu is on or not. > > > > Best, > > Karthik. > > > > > > *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Date: *Friday, 8 October 2021 at 14:55 > *To: *Mark Adams > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Thanks Mark, I will try your recommendations. > > Should I also change -dm_vec_type to hypre currently I have it as mpicuda? > > > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 14:33 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Hypre does not record its flops with PETSc's timers. > > Configure with and without CUDA and see if the timings change in PCApply. > > Hypre does not dynamically switch between CUDA and CPU solves at > this time, but you want to use -dm_mat_type hypre. > > Mark > > > > On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have > attached the python configuration file and -log_view output from running > the below command options > > > > mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 > -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type > hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 > -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor > > > > The problem was solved and converged but from the output file I suspect > hypre is not running on gpus as PCApply and DMCreate does *not* record > any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are > running on gpus. > > > > Can you please let me know if I need to add any extra flag to the attached > arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? > > > > Thanks, > > Karthik. > > > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > > Stefano > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 22 09:32:15 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 22 Nov 2021 15:32:15 +0000 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: Don?t use petsc-3.16.0 instead try main? Sure, no problem. I am can try. Best, Karthik. From: Matthew Knepley Date: Monday, 22 November 2021 at 15:26 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" Cc: Stefano Zampini , "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] hypre on gpus On Mon, Nov 22, 2021 at 10:20 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thanks Stefano, I just did a clean build with no problems. Can you try a clean build of main? Thanks, Matt I have another build without the --download-hypre-commit=origin/hypre_petsc but that gives a different error. [kchockalingam at glados tutorials]$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HYPRE_IJMatrixAssemble(): error code 12 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.15.3, Aug 06, 2021 [0]PETSC ERROR: ./ex4 on a named glados.dl.ac.uk by kchockalingam Mon Nov 22 15:07:46 2021 [0]PETSC ERROR: Configure options --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer" FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1 --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --with-debugging=no PETSC_ARCH=arch-ci-linux-cuda11-double [0]PETSC ERROR: #1 MatAssemblyEnd_HYPRE() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/impls/hypre/mhypre.c:1212 [0]PETSC ERROR: #2 MatAssemblyEnd() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/interface/matrix.c:5652 [0]PETSC ERROR: #3 main() at ex4.c:84 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -ksp_view [0]PETSC ERROR: -mat_type hypre [0]PETSC ERROR: -pc_type hypre [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- Best, Karthik. From: Stefano Zampini > Date: Monday, 22 November 2021 at 14:46 To: Matthew Knepley > Cc: "Chockalingam, Karthikeyan (STFC,DL,HC)" >, "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus You don't need to specify the HYPRE commit. Remove --download-hypre-commit=origin/hypre_petsc from the configuration options Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley > ha scritto: On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, Below is the entire error message: I cannot reproduce this: main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre KSP Object: 1 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 1 MPI processes type: hypre rows=56, cols=56 Norm of error 8.69801e-05 iterations 2 This is on the 'main' branch. So either there is some bug in release, or something is strange on your end. Since we run Hypre tests for the CI, I am leaning toward the latter. Can you try the 'main' branch? We will have to use this anyway if we want any fixes. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 11:33:41 2021 [0]PETSC ERROR: Configure options --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 --with-debugging=yes --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 [0]PETSC ERROR: #1 MatGetOwnershipRange() at /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 [0]PETSC ERROR: #2 main() at ex4.c:40 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -ksp_view [0]PETSC ERROR: -mat_type hypre [0]PETSC ERROR: -pc_type hypre [0]PETSC ERROR: -use_gpu_aware_mpi 0 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- I have also attached the make.log. Thank you for having a look. Best, Karthik. From: Matthew Knepley > Date: Monday, 22 November 2021 at 13:41 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: Mark Adams >, "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thank you for your response. I tried to run the same example petsc/src/ksp/ksp/tutorials$ ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre but it crashed with the below error [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() Hi Karthik, Please do not clip the error message. We need the entire output. This seems strange that you would get a logic error, since that should be the same on any architecture. So could you also send the make.log? Thanks, Matt Below are the options used to configure hypre with cuda support. Do you spot any mistakes? --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 Best, Karthik. From: Mark Adams > Date: Friday, 19 November 2021 at 16:31 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus You should run with -options_left to check that your options are being used. It may be -mat_type hypre. I have tested this: petsc/src/ksp/ksp/tutorials$ srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre You can add -log_view and that will print performance data for each method like KSPSolve. If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU. In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre. I saw that the flops were 100% GPU in hypre. On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view I profiled the run using NSYS - attached you find all the relevant files. Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active. I don?t see any cuda kernel relevant to solve. Is my assessment correct? How can I verify if hypre indeed is running on gpus? Best, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 18:47 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus I think you would want to use 'cuda' vec_type, but I . You might ask Hypre how one verifies that the GPU is used. Mark On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI > wrote: Yes, I used it for both cpu and gpu. Is that not okay? For gpu: -dm_mat_type hypre -dm_vec_type mpicuda For cpu: -dm_mat_type hypre -dm_vec_type mpi From: Mark Adams > Date: Friday, 8 October 2021 at 18:28 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Did you use -dm_mat_type hypre on the GPU case ? On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI > wrote: I tried a different exercise ran the same problem on two cpu cores and on two gpu: On gpu PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0 and on cpu PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00. I am not sure if that is any indication if hypre ran on gpus? Thanks, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 16:36 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI > wrote: The PCApply timing on gpu PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 and cpu PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 You don't have GPUs. probably. Use -dm_mat_type hypre. are close. It is hard for me tell if hypre on gpu is on or not. Best, Karthik. From: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Date: Friday, 8 October 2021 at 14:55 To: Mark Adams > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Thanks Mark, I will try your recommendations. Should I also change -dm_vec_type to hypre currently I have it as mpicuda? Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 14:33 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Hypre does not record its flops with PETSc's timers. Configure with and without CUDA and see if the timings change in PCApply. Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use -dm_mat_type hypre. Mark On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does not record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus. Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? Thanks, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.pelletier at protonmail.com Mon Nov 22 09:36:13 2021 From: milan.pelletier at protonmail.com (Milan Pelletier) Date: Mon, 22 Nov 2021 15:36:13 +0000 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: Matt, Thanks for the answer. I could fix my issue regarding unbalanced events - and so SNES ex5 does work well too. By the way, such imbalance does not trigger errors when log_view is using plain ascii output, but it does fail when ascii_xml is used instead. Unfortunately, regarding absolute paths, I had also tried to use cygwin paths (`/cygdrive/c/...`) but that does not work either. Best, Milan ??????? Original Message ??????? Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley a ?crit : > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users wrote: > >> Dear PETSc team, >> >> The way some options are passed to PETSc can be an issue on Windows, in particular regarding the "log_view" option. >> Since the colon ':' character is used as a separator, I do not see how absolute paths should be provided on Windows (since the drive name includes a colon). Is there a trick / workaround to overcome this? > > I thought we used Cygwin paths, but maybe one of the Windows people knows better. > >> Besides, when I try to use the ascii_xml format, it crashes in file xmllogevent.c, on line 751: >> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; >> (since nestedEvents is NULL). > > This likely means that you have unbalanced events. I can run this on examples. For instance does SNES ex5 work for you? > > Thanks, > > Matt > >> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). >> >> Thanks for your help, >> Best regards, >> Milan Pelletier > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > [https://www.cse.buffalo.edu/~knepley/](http://www.cse.buffalo.edu/~knepley/) -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 22 10:05:42 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 11:05:42 -0500 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < milan.pelletier at protonmail.com> wrote: > Matt, > Thanks for the answer. I could fix my issue regarding unbalanced events - > and so SNES ex5 does work well too. > By the way, such imbalance does not trigger errors when log_view is using > plain ascii output, but it does fail when ascii_xml is used instead. > Yes, we have a "failsafe" for the regular logging, but the XML is specifically hierarchical and we do not know how to close it. We could take another look at that. > Unfortunately, regarding absolute paths, I had also tried to use cygwin > paths (`/cygdrive/c/...`) but that does not work either. > Hmm, that should work I think. Satish, do we have a machine to login and test something like this? Thanks, Matt > Best, > Milan > > ??????? Original Message ??????? > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley > a ?crit : > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > petsc-users at mcs.anl.gov> wrote: > >> Dear PETSc team, >> >> The way some options are passed to PETSc can be an issue on Windows, in >> particular regarding the "log_view" option. >> Since the colon ':' character is used as a separator, I do not see how >> absolute paths should be provided on Windows (since the drive name includes >> a colon). Is there a trick / workaround to overcome this? >> > > I thought we used Cygwin paths, but maybe one of the Windows people knows > better. > > >> Besides, when I try to use the ascii_xml format, it crashes in file >> xmllogevent.c, on line 751: >> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; >> (since nestedEvents is NULL). >> > > This likely means that you have unbalanced events. I can run this on > examples. For instance does SNES ex5 work for you? > > Thanks, > > Matt > > >> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). >> >> Thanks for your help, >> Best regards, >> Milan Pelletier >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Nov 22 10:16:19 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 22 Nov 2021 11:16:19 -0500 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: <4CEAFDE2-F62E-46C5-95A3-9C67420AD878@petsc.dev> > On Nov 22, 2021, at 10:36 AM, Milan Pelletier via petsc-users wrote: > > Matt, > Thanks for the answer. I could fix my issue regarding unbalanced events - and so SNES ex5 does work well too. > By the way, such imbalance does not trigger errors when log_view is using plain ascii output, but it does fail when ascii_xml is used instead. > > Unfortunately, regarding absolute paths, I had also tried to use cygwin paths (`/cygdrive/c/...`) but that does not work either. What is the symptom of the failure, could you provide all the output printed and tell us what goes wrong. > > Best, > Milan > > ??????? Original Message ??????? > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley a ?crit : >> On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users > wrote: >> Dear PETSc team, >> >> The way some options are passed to PETSc can be an issue on Windows, in particular regarding the "log_view" option. >> Since the colon ':' character is used as a separator, I do not see how absolute paths should be provided on Windows (since the drive name includes a colon). Is there a trick / workaround to overcome this? >> >> I thought we used Cygwin paths, but maybe one of the Windows people knows better. >> >> Besides, when I try to use the ascii_xml format, it crashes in file xmllogevent.c, on line 751: >> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; >> (since nestedEvents is NULL). >> >> This likely means that you have unbalanced events. I can run this on examples. For instance does SNES ex5 work for you? >> >> Thanks, >> >> Matt >> >> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). >> >> Thanks for your help, >> Best regards, >> Milan Pelletier >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Nov 22 10:19:10 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 22 Nov 2021 10:19:10 -0600 (CST) Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: Well our build tools use cygwin - but the library is built with MS/Intel compilers - they use paths in windows notation. I'm not sure if there is a way to get windows PATHs without ':' Satish On Mon, 22 Nov 2021, Matthew Knepley wrote: > On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < > milan.pelletier at protonmail.com> wrote: > > > Matt, > > Thanks for the answer. I could fix my issue regarding unbalanced events - > > and so SNES ex5 does work well too. > > By the way, such imbalance does not trigger errors when log_view is using > > plain ascii output, but it does fail when ascii_xml is used instead. > > > > Yes, we have a "failsafe" for the regular logging, but the XML is > specifically hierarchical and we do not know how to close it. We could take > another look at that. > > > > Unfortunately, regarding absolute paths, I had also tried to use cygwin > > paths (`/cygdrive/c/...`) but that does not work either. > > > > Hmm, that should work I think. Satish, do we have a machine to login and > test something like this? > > Thanks, > > Matt > > > > Best, > > Milan > > > > ??????? Original Message ??????? > > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley > > a ?crit : > > > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > > petsc-users at mcs.anl.gov> wrote: > > > >> Dear PETSc team, > >> > >> The way some options are passed to PETSc can be an issue on Windows, in > >> particular regarding the "log_view" option. > >> Since the colon ':' character is used as a separator, I do not see how > >> absolute paths should be provided on Windows (since the drive name includes > >> a colon). Is there a trick / workaround to overcome this? > >> > > > > I thought we used Cygwin paths, but maybe one of the Windows people knows > > better. > > > > > >> Besides, when I try to use the ascii_xml format, it crashes in file > >> xmllogevent.c, on line 751: > >> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > >> (since nestedEvents is NULL). > >> > > > > This likely means that you have unbalanced events. I can run this on > > examples. For instance does SNES ex5 work for you? > > > > Thanks, > > > > Matt > > > > > >> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > >> > >> Thanks for your help, > >> Best regards, > >> Milan Pelletier > >> > >> > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > From milan.pelletier at protonmail.com Mon Nov 22 10:29:53 2021 From: milan.pelletier at protonmail.com (Milan Pelletier) Date: Mon, 22 Nov 2021 16:29:53 +0000 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: I get a "Unable to open file" error: "Cannot open PetscViewer file: \\cygdrive\\c\\some_path_that_exists\\petsc_log.txt". My input to PETSc was "-log_view /cygdrive/c/some_path_that_exists/petsc_log.txt" so the path gets converted into windows "\\" separators. I tried and looked for a way to avoid using a colon but could not find one. Milan ??????? Original Message ??????? Le lundi 22 novembre 2021 ? 5:05 PM, Matthew Knepley a ?crit : > On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier wrote: > >> Matt, >> Thanks for the answer. I could fix my issue regarding unbalanced events - and so SNES ex5 does work well too. >> By the way, such imbalance does not trigger errors when log_view is using plain ascii output, but it does fail when ascii_xml is used instead. > > Yes, we have a "failsafe" for the regular logging, but the XML is specifically hierarchical and we do not know how to close it. We could take another look at that. > >> Unfortunately, regarding absolute paths, I had also tried to use cygwin paths (`/cygdrive/c/...`) but that does not work either. > > Hmm, that should work I think. Satish, do we have a machine to login and test something like this? > > Thanks, > > Matt > >> Best, >> Milan >> >> ??????? Original Message ??????? >> Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley a ?crit : >> >>> On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users wrote: >>> >>>> Dear PETSc team, >>>> >>>> The way some options are passed to PETSc can be an issue on Windows, in particular regarding the "log_view" option. >>>> Since the colon ':' character is used as a separator, I do not see how absolute paths should be provided on Windows (since the drive name includes a colon). Is there a trick / workaround to overcome this? >>> >>> I thought we used Cygwin paths, but maybe one of the Windows people knows better. >>> >>>> Besides, when I try to use the ascii_xml format, it crashes in file xmllogevent.c, on line 751: >>>> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; >>>> (since nestedEvents is NULL). >>> >>> This likely means that you have unbalanced events. I can run this on examples. For instance does SNES ex5 work for you? >>> >>> Thanks, >>> >>> Matt >>> >>>> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). >>>> >>>> Thanks for your help, >>>> Best regards, >>>> Milan Pelletier >>> >>> -- >>> >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> [https://www.cse.buffalo.edu/~knepley/](http://www.cse.buffalo.edu/~knepley/) > > -- > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > [https://www.cse.buffalo.edu/~knepley/](http://www.cse.buffalo.edu/~knepley/) -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Nov 22 10:36:26 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 22 Nov 2021 10:36:26 -0600 (CST) Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: I think one use case where ':' can be avoided with absolute path is - - if currently on the same drive Satish ---- C:\cygwin64\home\balay>dir \ Volume in drive C is OS Volume Serial Number is 30F3-283B Directory of C:\ 05/11/2021 07:19 AM cygwin64 04/02/2021 06:29 PM dell 11/22/2021 07:59 AM Intel 12/07/2019 01:14 AM PerfLogs 11/03/2021 06:03 AM Program Files 04/10/2021 08:11 AM Program Files (x86) 09/10/2021 01:24 PM Users 11/09/2021 07:13 PM Windows 0 File(s) 0 bytes 8 Dir(s) 87,535,738,880 bytes free C:\cygwin64\home\balay>dir \cygwin64 Volume in drive C is OS Volume Serial Number is 30F3-283B Directory of C:\cygwin64 05/11/2021 07:19 AM . 05/11/2021 07:19 AM .. 09/10/2021 12:24 PM bin 04/02/2021 05:45 PM 53,342 Cygwin-Terminal.ico 05/11/2021 07:19 AM 88 Cygwin.bat 04/02/2021 05:45 PM 157,097 Cygwin.ico 04/02/2021 05:45 PM dev 09/10/2021 12:23 PM etc 09/10/2021 01:24 PM home 09/10/2021 12:23 PM lib 04/02/2021 05:45 PM sbin 11/22/2021 08:33 AM tmp 09/10/2021 12:23 PM usr 04/02/2021 07:40 PM var 3 File(s) 210,527 bytes 11 Dir(s) 87,535,652,864 bytes free C:\cygwin64\home\balay> On Mon, 22 Nov 2021, Satish Balay via petsc-users wrote: > Well our build tools use cygwin - but the library is built with MS/Intel compilers - they use paths in windows notation. > > I'm not sure if there is a way to get windows PATHs without ':' > > Satish > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > > On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < > > milan.pelletier at protonmail.com> wrote: > > > > > Matt, > > > Thanks for the answer. I could fix my issue regarding unbalanced events - > > > and so SNES ex5 does work well too. > > > By the way, such imbalance does not trigger errors when log_view is using > > > plain ascii output, but it does fail when ascii_xml is used instead. > > > > > > > Yes, we have a "failsafe" for the regular logging, but the XML is > > specifically hierarchical and we do not know how to close it. We could take > > another look at that. > > > > > > > Unfortunately, regarding absolute paths, I had also tried to use cygwin > > > paths (`/cygdrive/c/...`) but that does not work either. > > > > > > > Hmm, that should work I think. Satish, do we have a machine to login and > > test something like this? > > > > Thanks, > > > > Matt > > > > > > > Best, > > > Milan > > > > > > ??????? Original Message ??????? > > > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley > > > a ?crit : > > > > > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > > > petsc-users at mcs.anl.gov> wrote: > > > > > >> Dear PETSc team, > > >> > > >> The way some options are passed to PETSc can be an issue on Windows, in > > >> particular regarding the "log_view" option. > > >> Since the colon ':' character is used as a separator, I do not see how > > >> absolute paths should be provided on Windows (since the drive name includes > > >> a colon). Is there a trick / workaround to overcome this? > > >> > > > > > > I thought we used Cygwin paths, but maybe one of the Windows people knows > > > better. > > > > > > > > >> Besides, when I try to use the ascii_xml format, it crashes in file > > >> xmllogevent.c, on line 751: > > >> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > > >> (since nestedEvents is NULL). > > >> > > > > > > This likely means that you have unbalanced events. I can run this on > > > examples. For instance does SNES ex5 work for you? > > > > > > Thanks, > > > > > > Matt > > > > > > > > >> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > > >> > > >> Thanks for your help, > > >> Best regards, > > >> Milan Pelletier > > >> > > >> > > > > > > -- > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to which their > > > experiments lead. > > > -- Norbert Wiener > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > > > > > > > > From milan.pelletier at protonmail.com Mon Nov 22 10:36:22 2021 From: milan.pelletier at protonmail.com (Milan Pelletier) Date: Mon, 22 Nov 2021 16:36:22 +0000 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: Sorry, my input was actually "-log_view :/cygdrive/c/some_path_that_exists/petsc_log.txt" with the proper colon separator required by PETSc! Milan Pelletier ??????? Original Message ??????? Le lundi 22 novembre 2021 ? 5:29 PM, Milan Pelletier via petsc-users a ?crit : > I get a "Unable to open file" error: > "Cannot open PetscViewer file: \\cygdrive\\c\\some_path_that_exists\\petsc_log.txt". > > My input to PETSc was "-log_view /cygdrive/c/some_path_that_exists/petsc_log.txt" so the path gets converted into windows "\\" separators. > > I tried and looked for a way to avoid using a colon but could not find one. > Milan > > ??????? Original Message ??????? > Le lundi 22 novembre 2021 ? 5:05 PM, Matthew Knepley a ?crit : > >> On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier wrote: >> >>> Matt, >>> Thanks for the answer. I could fix my issue regarding unbalanced events - and so SNES ex5 does work well too. >>> By the way, such imbalance does not trigger errors when log_view is using plain ascii output, but it does fail when ascii_xml is used instead. >> >> Yes, we have a "failsafe" for the regular logging, but the XML is specifically hierarchical and we do not know how to close it. We could take another look at that. >> >>> Unfortunately, regarding absolute paths, I had also tried to use cygwin paths (`/cygdrive/c/...`) but that does not work either. >> >> Hmm, that should work I think. Satish, do we have a machine to login and test something like this? >> >> Thanks, >> >> Matt >> >>> Best, >>> Milan >>> >>> ??????? Original Message ??????? >>> Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley a ?crit : >>> >>>> On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users wrote: >>>> >>>>> Dear PETSc team, >>>>> >>>>> The way some options are passed to PETSc can be an issue on Windows, in particular regarding the "log_view" option. >>>>> Since the colon ':' character is used as a separator, I do not see how absolute paths should be provided on Windows (since the drive name includes a colon). Is there a trick / workaround to overcome this? >>>> >>>> I thought we used Cygwin paths, but maybe one of the Windows people knows better. >>>> >>>>> Besides, when I try to use the ascii_xml format, it crashes in file xmllogevent.c, on line 751: >>>>> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; >>>>> (since nestedEvents is NULL). >>>> >>>> This likely means that you have unbalanced events. I can run this on examples. For instance does SNES ex5 work for you? >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>>> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). >>>>> >>>>> Thanks for your help, >>>>> Best regards, >>>>> Milan Pelletier >>>> >>>> -- >>>> >>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>> -- Norbert Wiener >>>> >>>> [https://www.cse.buffalo.edu/~knepley/](http://www.cse.buffalo.edu/~knepley/) >> >> -- >> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> [https://www.cse.buffalo.edu/~knepley/](http://www.cse.buffalo.edu/~knepley/) -------------- next part -------------- An HTML attachment was scrubbed... URL: From milan.pelletier at protonmail.com Mon Nov 22 10:49:18 2021 From: milan.pelletier at protonmail.com (Milan Pelletier) Date: Mon, 22 Nov 2021 16:49:18 +0000 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: Satish, Indeed, but I was trying to write it to another drive. Anyways, that's not a big deal, only inconvenience. I can just use your trick and create a symlink to point to the proper location on another drive. Thank you all for your help, Best, Milan Pelletier ??????? Original Message ??????? Le lundi 22 novembre 2021 ? 5:36 PM, Satish Balay via petsc-users a ?crit : > I think one use case where ':' can be avoided with absolute path is - - if currently on the same drive > > Satish > > --------------------------------------------------------------------------------------------------------------- > > C:\cygwin64\home\balay>dir \ > > Volume in drive C is OS > > Volume Serial Number is 30F3-283B > > Directory of C:\ > > 05/11/2021 07:19 AM cygwin64 > > 04/02/2021 06:29 PM dell > > 11/22/2021 07:59 AM Intel > > 12/07/2019 01:14 AM PerfLogs > > 11/03/2021 06:03 AM Program Files > > 04/10/2021 08:11 AM Program Files (x86) > > 09/10/2021 01:24 PM Users > > 11/09/2021 07:13 PM Windows > > 0 File(s) 0 bytes > 8 Dir(s) 87,535,738,880 bytes free > > > C:\cygwin64\home\balay>dir \cygwin64 > > Volume in drive C is OS > > Volume Serial Number is 30F3-283B > > Directory of C:\cygwin64 > > 05/11/2021 07:19 AM . > > 05/11/2021 07:19 AM .. > > 09/10/2021 12:24 PM bin > > 04/02/2021 05:45 PM 53,342 Cygwin-Terminal.ico > > 05/11/2021 07:19 AM 88 Cygwin.bat > > 04/02/2021 05:45 PM 157,097 Cygwin.ico > > 04/02/2021 05:45 PM dev > > 09/10/2021 12:23 PM etc > > 09/10/2021 01:24 PM home > > 09/10/2021 12:23 PM lib > > 04/02/2021 05:45 PM sbin > > 11/22/2021 08:33 AM tmp > > 09/10/2021 12:23 PM usr > > 04/02/2021 07:40 PM var > > 3 File(s) 210,527 bytes > 11 Dir(s) 87,535,652,864 bytes free > > > C:\cygwin64\home\balay> > > On Mon, 22 Nov 2021, Satish Balay via petsc-users wrote: > > > Well our build tools use cygwin - but the library is built with MS/Intel compilers - they use paths in windows notation. > > > > I'm not sure if there is a way to get windows PATHs without ':' > > > > Satish > > > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > > > > On Mon, Nov 22, 2021 at 10:36 AM > > > > > > Milan Pelletier > > > > > > < > > > > > > milan.pelletier at protonmail.com> wrote: > > > > > > > Matt, > > > > > > > > Thanks for the answer. I could fix my issue regarding unbalanced events - > > > > > > > > and so SNES ex5 does work well too. > > > > > > > > By the way, such imbalance does not trigger errors when log_view is using > > > > > > > > plain ascii output, but it does fail when ascii_xml is used instead. > > > > > > Yes, we have a "failsafe" for the regular logging, but the XML is > > > > > > specifically hierarchical and we do not know how to close it. We could take > > > > > > another look at that. > > > > > > > Unfortunately, regarding absolute paths, I had also tried to use cygwin > > > > > > > > paths (`/cygdrive/c/...`) but that does not work either. > > > > > > Hmm, that should work I think. Satish, do we have a machine to login and > > > > > > test something like this? > > > > > > Thanks, > > > > > > Matt > > > > > > > > > > Best, > > > > > > > > Milan > > > > > > > > ??????? Original Message ??????? > > > > > > > > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley knepley at gmail.com > > > > > > > > a ?crit : > > > > > > > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > > > > > > > > petsc-users at mcs.anl.gov> wrote: > > > > > > > > > Dear PETSc team, > > > > > > > > > > The way some options are passed to PETSc can be an issue on Windows, in > > > > > > > > > > particular regarding the "log_view" option. > > > > > > > > > > Since the colon ':' character is used as a separator, I do not see how > > > > > > > > > > absolute paths should be provided on Windows (since the drive name includes > > > > > > > > > > a colon). Is there a trick / workaround to overcome this? > > > > > > > > I thought we used Cygwin paths, but maybe one of the Windows people knows > > > > > > > > better. > > > > > > > > > Besides, when I try to use the ascii_xml format, it crashes in file > > > > > > > > > > xmllogevent.c, on line 751: > > > > > > > > > > illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > > > > > > > > > > (since nestedEvents is NULL). > > > > > > > > This likely means that you have unbalanced events. I can run this on > > > > > > > > examples. For instance does SNES ex5 work for you? > > > > > > > > Thanks, > > > > > > > > Matt > > > > > > > > > > > > > I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > > > > > > > > > > Thanks for your help, > > > > > > > > > > Best regards, > > > > > > > > > > Milan Pelletier > > > > > > > > -- > > > > > > > > What most experimenters take for granted before they begin their > > > > > > > > experiments is infinitely more interesting than any results to which their > > > > > > > > experiments lead. > > > > > > > > -- Norbert Wiener > > > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > http://www.cse.buffalo.edu/~knepley/ From balay at mcs.anl.gov Mon Nov 22 10:52:06 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 22 Nov 2021 10:52:06 -0600 (CST) Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: Matt, Perhaps we should also support a different char other than ':' as a separator. Since most special chars are interpreted by shell in some form or the other - its not clear to me what other separator chars we can use. Satish On Mon, 22 Nov 2021, Milan Pelletier wrote: > Satish, > Indeed, but I was trying to write it to another drive. > Anyways, that's not a big deal, only inconvenience. I can just use your trick and create a symlink to point to the proper location on another drive. > > Thank you all for your help, > > Best, > Milan Pelletier > > ??????? Original Message ??????? > > Le lundi 22 novembre 2021 ? 5:36 PM, Satish Balay via petsc-users a ?crit : > > > I think one use case where ':' can be avoided with absolute path is - - if currently on the same drive > > > > Satish > > > > --------------------------------------------------------------------------------------------------------------- > > > > C:\cygwin64\home\balay>dir \ > > > > Volume in drive C is OS > > > > Volume Serial Number is 30F3-283B > > > > Directory of C:\ > > > > 05/11/2021 07:19 AM cygwin64 > > > > 04/02/2021 06:29 PM dell > > > > 11/22/2021 07:59 AM Intel > > > > 12/07/2019 01:14 AM PerfLogs > > > > 11/03/2021 06:03 AM Program Files > > > > 04/10/2021 08:11 AM Program Files (x86) > > > > 09/10/2021 01:24 PM Users > > > > 11/09/2021 07:13 PM Windows > > > > 0 File(s) 0 bytes > > 8 Dir(s) 87,535,738,880 bytes free > > > > > > C:\cygwin64\home\balay>dir \cygwin64 > > > > Volume in drive C is OS > > > > Volume Serial Number is 30F3-283B > > > > Directory of C:\cygwin64 > > > > 05/11/2021 07:19 AM . > > > > 05/11/2021 07:19 AM .. > > > > 09/10/2021 12:24 PM bin > > > > 04/02/2021 05:45 PM 53,342 Cygwin-Terminal.ico > > > > 05/11/2021 07:19 AM 88 Cygwin.bat > > > > 04/02/2021 05:45 PM 157,097 Cygwin.ico > > > > 04/02/2021 05:45 PM dev > > > > 09/10/2021 12:23 PM etc > > > > 09/10/2021 01:24 PM home > > > > 09/10/2021 12:23 PM lib > > > > 04/02/2021 05:45 PM sbin > > > > 11/22/2021 08:33 AM tmp > > > > 09/10/2021 12:23 PM usr > > > > 04/02/2021 07:40 PM var > > > > 3 File(s) 210,527 bytes > > 11 Dir(s) 87,535,652,864 bytes free > > > > > > C:\cygwin64\home\balay> > > > > On Mon, 22 Nov 2021, Satish Balay via petsc-users wrote: > > > > > Well our build tools use cygwin - but the library is built with MS/Intel compilers - they use paths in windows notation. > > > > > > I'm not sure if there is a way to get windows PATHs without ':' > > > > > > Satish > > > > > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > > > > > > On Mon, Nov 22, 2021 at 10:36 AM > > > > > > > > Milan Pelletier > > > > > > > > < > > > > > > > > milan.pelletier at protonmail.com> wrote: > > > > > > > > > Matt, > > > > > > > > > > Thanks for the answer. I could fix my issue regarding unbalanced events - > > > > > > > > > > and so SNES ex5 does work well too. > > > > > > > > > > By the way, such imbalance does not trigger errors when log_view is using > > > > > > > > > > plain ascii output, but it does fail when ascii_xml is used instead. > > > > > > > > Yes, we have a "failsafe" for the regular logging, but the XML is > > > > > > > > specifically hierarchical and we do not know how to close it. We could take > > > > > > > > another look at that. > > > > > > > > > Unfortunately, regarding absolute paths, I had also tried to use cygwin > > > > > > > > > > paths (`/cygdrive/c/...`) but that does not work either. > > > > > > > > Hmm, that should work I think. Satish, do we have a machine to login and > > > > > > > > test something like this? > > > > > > > > Thanks, > > > > > > > > Matt > > > > > > > > > > > > > Best, > > > > > > > > > > Milan > > > > > > > > > > ??????? Original Message ??????? > > > > > > > > > > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley knepley at gmail.com > > > > > > > > > > a ?crit : > > > > > > > > > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > > > > > > > > > > petsc-users at mcs.anl.gov> wrote: > > > > > > > > > > > Dear PETSc team, > > > > > > > > > > > > The way some options are passed to PETSc can be an issue on Windows, in > > > > > > > > > > > > particular regarding the "log_view" option. > > > > > > > > > > > > Since the colon ':' character is used as a separator, I do not see how > > > > > > > > > > > > absolute paths should be provided on Windows (since the drive name includes > > > > > > > > > > > > a colon). Is there a trick / workaround to overcome this? > > > > > > > > > > I thought we used Cygwin paths, but maybe one of the Windows people knows > > > > > > > > > > better. > > > > > > > > > > > Besides, when I try to use the ascii_xml format, it crashes in file > > > > > > > > > > > > xmllogevent.c, on line 751: > > > > > > > > > > > > illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > > > > > > > > > > > > (since nestedEvents is NULL). > > > > > > > > > > This likely means that you have unbalanced events. I can run this on > > > > > > > > > > examples. For instance does SNES ex5 work for you? > > > > > > > > > > Thanks, > > > > > > > > > > Matt > > > > > > > > > > > > > > > > I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > > > > > > > > > > > > Thanks for your help, > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Milan Pelletier > > > > > > > > > > -- > > > > > > > > > > What most experimenters take for granted before they begin their > > > > > > > > > > experiments is infinitely more interesting than any results to which their > > > > > > > > > > experiments lead. > > > > > > > > > > -- Norbert Wiener > > > > > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > http://www.cse.buffalo.edu/~knepley/ > From bsmith at petsc.dev Mon Nov 22 10:57:34 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 22 Nov 2021 11:57:34 -0500 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: Message-ID: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> What does Windows use for a path separator? > On Nov 22, 2021, at 11:19 AM, Satish Balay via petsc-users wrote: > > Well our build tools use cygwin - but the library is built with MS/Intel compilers - they use paths in windows notation. > > I'm not sure if there is a way to get windows PATHs without ':' > > Satish > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > >> On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < >> milan.pelletier at protonmail.com> wrote: >> >>> Matt, >>> Thanks for the answer. I could fix my issue regarding unbalanced events - >>> and so SNES ex5 does work well too. >>> By the way, such imbalance does not trigger errors when log_view is using >>> plain ascii output, but it does fail when ascii_xml is used instead. >>> >> >> Yes, we have a "failsafe" for the regular logging, but the XML is >> specifically hierarchical and we do not know how to close it. We could take >> another look at that. >> >> >>> Unfortunately, regarding absolute paths, I had also tried to use cygwin >>> paths (`/cygdrive/c/...`) but that does not work either. >>> >> >> Hmm, that should work I think. Satish, do we have a machine to login and >> test something like this? >> >> Thanks, >> >> Matt >> >> >>> Best, >>> Milan >>> >>> ??????? Original Message ??????? >>> Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley >>> a ?crit : >>> >>> On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < >>> petsc-users at mcs.anl.gov> wrote: >>> >>>> Dear PETSc team, >>>> >>>> The way some options are passed to PETSc can be an issue on Windows, in >>>> particular regarding the "log_view" option. >>>> Since the colon ':' character is used as a separator, I do not see how >>>> absolute paths should be provided on Windows (since the drive name includes >>>> a colon). Is there a trick / workaround to overcome this? >>>> >>> >>> I thought we used Cygwin paths, but maybe one of the Windows people knows >>> better. >>> >>> >>>> Besides, when I try to use the ascii_xml format, it crashes in file >>>> xmllogevent.c, on line 751: >>>> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; >>>> (since nestedEvents is NULL). >>>> >>> >>> This likely means that you have unbalanced events. I can run this on >>> examples. For instance does SNES ex5 work for you? >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). >>>> >>>> Thanks for your help, >>>> Best regards, >>>> Milan Pelletier >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Nov 22 11:00:10 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 22 Nov 2021 11:00:10 -0600 (CST) Subject: [petsc-users] Issues with log_view on windows In-Reply-To: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> References: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> Message-ID: On Mon, 22 Nov 2021, Barry Smith wrote: > > What does Windows use for a path separator? '\' For ex: C:\cygwin64\home\balay Satish > > > On Nov 22, 2021, at 11:19 AM, Satish Balay via petsc-users wrote: > > > > Well our build tools use cygwin - but the library is built with MS/Intel compilers - they use paths in windows notation. > > > > I'm not sure if there is a way to get windows PATHs without ':' > > > > Satish > > > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > > >> On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < > >> milan.pelletier at protonmail.com> wrote: > >> > >>> Matt, > >>> Thanks for the answer. I could fix my issue regarding unbalanced events - > >>> and so SNES ex5 does work well too. > >>> By the way, such imbalance does not trigger errors when log_view is using > >>> plain ascii output, but it does fail when ascii_xml is used instead. > >>> > >> > >> Yes, we have a "failsafe" for the regular logging, but the XML is > >> specifically hierarchical and we do not know how to close it. We could take > >> another look at that. > >> > >> > >>> Unfortunately, regarding absolute paths, I had also tried to use cygwin > >>> paths (`/cygdrive/c/...`) but that does not work either. > >>> > >> > >> Hmm, that should work I think. Satish, do we have a machine to login and > >> test something like this? > >> > >> Thanks, > >> > >> Matt > >> > >> > >>> Best, > >>> Milan > >>> > >>> ??????? Original Message ??????? > >>> Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley > >>> a ?crit : > >>> > >>> On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > >>> petsc-users at mcs.anl.gov> wrote: > >>> > >>>> Dear PETSc team, > >>>> > >>>> The way some options are passed to PETSc can be an issue on Windows, in > >>>> particular regarding the "log_view" option. > >>>> Since the colon ':' character is used as a separator, I do not see how > >>>> absolute paths should be provided on Windows (since the drive name includes > >>>> a colon). Is there a trick / workaround to overcome this? > >>>> > >>> > >>> I thought we used Cygwin paths, but maybe one of the Windows people knows > >>> better. > >>> > >>> > >>>> Besides, when I try to use the ascii_xml format, it crashes in file > >>>> xmllogevent.c, on line 751: > >>>> illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > >>>> (since nestedEvents is NULL). > >>>> > >>> > >>> This likely means that you have unbalanced events. I can run this on > >>> examples. For instance does SNES ex5 work for you? > >>> > >>> Thanks, > >>> > >>> Matt > >>> > >>> > >>>> I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > >>>> > >>>> Thanks for your help, > >>>> Best regards, > >>>> Milan Pelletier > >>>> > >>>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their > >>> experiments is infinitely more interesting than any results to which their > >>> experiments lead. > >>> -- Norbert Wiener > >>> > >>> https://www.cse.buffalo.edu/~knepley/ > >>> > > > From bhargav.subramanya at kaust.edu.sa Mon Nov 22 11:09:11 2021 From: bhargav.subramanya at kaust.edu.sa (Bhargav Subramanya) Date: Mon, 22 Nov 2021 20:09:11 +0300 Subject: [petsc-users] dmplex extruded layers Message-ID: Dear All, I have a prismatic mesh generated by extruding a base icosahedron spherical surface mesh. The mesh is distributed on 2 processes after extrusion. The dof for my problem are defined only on the simplices of the spherical layers. 1. For a given spherical layer, I want to get points, which are basically simplices, lying on that layer only. DMPlexGetHeightStratum returns the points on all the spherical Layers. I can probably use DMPlexFindVertices (based on the radius of the spherical layer) and DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to retrieve the points (which are simplices) on a given spherical layer in the extruded mesh? 2. I am able to visualize the entire mesh using dm_view. Is there a way to get the mesh file for the local dm from a specific process? 3. One of my previous emails might have got lost as I mistakenly attached a large mesh file and sent it. So I am repeating the question here. DMPlexExtrude gives the following error after distributing the base 2D spherical mesh. Both refinement or/and extrusion after dm distribution does not work for me. In fact, I tried with src/dm/impls/plex/tutorials/ex10.c also. Although the mesh is distributed after the extrusion in ex10.c (which is working fine for me), I tried to distribute before extrusion, and I get the following error. Could you please suggest where I might be making a mistake? [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: This DMPlex is distributed but its PointSF has no graph set [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.16.1, unknown [0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named kl-21859 by subrambm Mon Nov 22 19:47:14 2021 [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-superlu_dist --download-hypre --download-fiat --download-generator --download-triangle --download-tetgen --download-chaco --download-make -download-boost --download-cmake --download-ml --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch --download-hdf5 --force [0]PETSC ERROR: #1 DMPlexCheckPointSF() at /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554 [0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595 [0]PETSC ERROR: #3 DMPlexInterpolate() at /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357 [0]PETSC ERROR: #4 DMPlexExtrude() at /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543 [0]PETSC ERROR: #5 CreateMesh() at ex10.c:161 [0]PETSC ERROR: #6 main() at ex10.c:180 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -dm_plex_extrude_layers 3 [0]PETSC ERROR: -dm_view vtk:mesh.vtk [0]PETSC ERROR: -init_dm_plex_dim 2 [0]PETSC ERROR: -petscpartitioner_type simple [0]PETSC ERROR: -srf_dm_distribute [0]PETSC ERROR: -srf_dm_refine 0 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- Thanks, Bhargav -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Mon Nov 22 11:41:54 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Mon, 22 Nov 2021 17:41:54 +0000 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: @Matthew Knepley Yes, it works with main build! @Mark Adams I have attached the log_view output of ex4 for your reference. But pcapply or pcsetup did not record any gpu flops. If it is not too much trouble can you send me the log_view output of ksp/ksp/tutorial/ex45 using hypre? Many thanks! Karthik. From: Matthew Knepley Date: Monday, 22 November 2021 at 15:26 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" Cc: Stefano Zampini , "petsc-users at mcs.anl.gov" Subject: Re: [petsc-users] hypre on gpus On Mon, Nov 22, 2021 at 10:20 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thanks Stefano, I just did a clean build with no problems. Can you try a clean build of main? Thanks, Matt I have another build without the --download-hypre-commit=origin/hypre_petsc but that gives a different error. [kchockalingam at glados tutorials]$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HYPRE_IJMatrixAssemble(): error code 12 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.15.3, Aug 06, 2021 [0]PETSC ERROR: ./ex4 on a named glados.dl.ac.uk by kchockalingam Mon Nov 22 15:07:46 2021 [0]PETSC ERROR: Configure options --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer" FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1 --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --with-debugging=no PETSC_ARCH=arch-ci-linux-cuda11-double [0]PETSC ERROR: #1 MatAssemblyEnd_HYPRE() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/impls/hypre/mhypre.c:1212 [0]PETSC ERROR: #2 MatAssemblyEnd() at /home/kchockalingam/tools/petsc-3.15.3/src/mat/interface/matrix.c:5652 [0]PETSC ERROR: #3 main() at ex4.c:84 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -ksp_view [0]PETSC ERROR: -mat_type hypre [0]PETSC ERROR: -pc_type hypre [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- Best, Karthik. From: Stefano Zampini > Date: Monday, 22 November 2021 at 14:46 To: Matthew Knepley > Cc: "Chockalingam, Karthikeyan (STFC,DL,HC)" >, "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus You don't need to specify the HYPRE commit. Remove --download-hypre-commit=origin/hypre_petsc from the configuration options Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley > ha scritto: On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hi Matt, Below is the entire error message: I cannot reproduce this: main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view -ksp_type cg -mat_type hypre -pc_type hypre KSP Object: 1 MPI processes type: cg maximum iterations=10000, initial guess is zero tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. left preconditioning using PRECONDITIONED norm type for convergence test PC Object: 1 MPI processes type: hypre HYPRE BoomerAMG preconditioning Cycle type V Maximum number of levels 25 Maximum number of iterations PER hypre call 1 Convergence tolerance PER hypre call 0. Threshold for strong coupling 0.25 Interpolation truncation factor 0. Interpolation: max elements per row 0 Number of levels of aggressive coarsening 0 Number of paths for aggressive coarsening 1 Maximum row sums 0.9 Sweeps down 1 Sweeps up 1 Sweeps on coarse 1 Relax down symmetric-SOR/Jacobi Relax up symmetric-SOR/Jacobi Relax on coarse Gaussian-elimination Relax weight (all) 1. Outer relax weight (all) 1. Using CF-relaxation Not using more complex smoothers. Measure type local Coarsen type Falgout Interpolation type classical linear system matrix = precond matrix: Mat Object: 1 MPI processes type: hypre rows=56, cols=56 Norm of error 8.69801e-05 iterations 2 This is on the 'main' branch. So either there is some bug in release, or something is strange on your end. Since we run Hypre tests for the CI, I am leaning toward the latter. Can you try the 'main' branch? We will have to use this anyway if we want any fixes. Thanks, Matt [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 11:33:41 2021 [0]PETSC ERROR: Configure options --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 --with-debugging=yes --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 [0]PETSC ERROR: #1 MatGetOwnershipRange() at /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 [0]PETSC ERROR: #2 main() at ex4.c:40 [0]PETSC ERROR: PETSc Option Table entries: [0]PETSC ERROR: -ksp_type cg [0]PETSC ERROR: -ksp_view [0]PETSC ERROR: -mat_type hypre [0]PETSC ERROR: -pc_type hypre [0]PETSC ERROR: -use_gpu_aware_mpi 0 [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint at mcs.anl.gov---------- -------------------------------------------------------------------------- I have also attached the make.log. Thank you for having a look. Best, Karthik. From: Matthew Knepley > Date: Monday, 22 November 2021 at 13:41 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: Mark Adams >, "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Thank you for your response. I tried to run the same example petsc/src/ksp/ksp/tutorials$ ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre but it crashed with the below error [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Object is in wrong state [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "mat" before MatGetOwnershipRange() Hi Karthik, Please do not clip the error message. We need the entire output. This seems strange that you would get a logic error, since that should be the same on any architecture. So could you also send the make.log? Thanks, Matt Below are the options used to configure hypre with cuda support. Do you spot any mistakes? --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx -with-fc=mpif90 Best, Karthik. From: Mark Adams > Date: Friday, 19 November 2021 at 16:31 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus You should run with -options_left to check that your options are being used. It may be -mat_type hypre. I have tested this: petsc/src/ksp/ksp/tutorials$ srun -n2 ./ex4 -ksp_type cg -mat_type hypre -ksp_view -pc_type hypre You can add -log_view and that will print performance data for each method like KSPSolve. If you are configured with a GPU there will be some extra columns that give the percent of flops on the GPU. In the past hypre has not registered flops with us, but I do get flops from hypre now, and -ksp_view showed that it did indeed use hypre. I saw that the flops were 100% GPU in hypre. On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view I profiled the run using NSYS - attached you find all the relevant files. Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active. I don?t see any cuda kernel relevant to solve. Is my assessment correct? How can I verify if hypre indeed is running on gpus? Best, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 18:47 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus I think you would want to use 'cuda' vec_type, but I . You might ask Hypre how one verifies that the GPU is used. Mark On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI > wrote: Yes, I used it for both cpu and gpu. Is that not okay? For gpu: -dm_mat_type hypre -dm_vec_type mpicuda For cpu: -dm_mat_type hypre -dm_vec_type mpi From: Mark Adams > Date: Friday, 8 October 2021 at 18:28 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Did you use -dm_mat_type hypre on the GPU case ? On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI > wrote: I tried a different exercise ran the same problem on two cpu cores and on two gpu: On gpu PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0 and on cpu PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00. I am not sure if that is any indication if hypre ran on gpus? Thanks, Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 16:36 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI > wrote: The PCApply timing on gpu PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0 and cpu PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 You don't have GPUs. probably. Use -dm_mat_type hypre. are close. It is hard for me tell if hypre on gpu is on or not. Best, Karthik. From: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Date: Friday, 8 October 2021 at 14:55 To: Mark Adams > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Thanks Mark, I will try your recommendations. Should I also change -dm_vec_type to hypre currently I have it as mpicuda? Karthik. From: Mark Adams > Date: Friday, 8 October 2021 at 14:33 To: "Chockalingam, Karthikeyan (STFC,DL,HC)" > Cc: "petsc-users at mcs.anl.gov" > Subject: Re: [petsc-users] hypre on gpus Hypre does not record its flops with PETSc's timers. Configure with and without CUDA and see if the timings change in PCApply. Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use -dm_mat_type hypre. Mark On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI > wrote: Hello, I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does not record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus. Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? Thanks, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -- Stefano -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: log_hypre_test.txt URL: From knepley at gmail.com Mon Nov 22 12:03:05 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 13:03:05 -0500 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> References: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> Message-ID: On Mon, Nov 22, 2021 at 11:57 AM Barry Smith wrote: > > What does Windows use for a path separator? > semicolon Maybe we could use either one as a separator, since semicolon is also used by the UNIX shell. Matt > On Nov 22, 2021, at 11:19 AM, Satish Balay via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Well our build tools use cygwin - but the library is built with MS/Intel > compilers - they use paths in windows notation. > > I'm not sure if there is a way to get windows PATHs without ':' > > Satish > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < > milan.pelletier at protonmail.com> wrote: > > Matt, > Thanks for the answer. I could fix my issue regarding unbalanced events - > and so SNES ex5 does work well too. > By the way, such imbalance does not trigger errors when log_view is using > plain ascii output, but it does fail when ascii_xml is used instead. > > > Yes, we have a "failsafe" for the regular logging, but the XML is > specifically hierarchical and we do not know how to close it. We could take > another look at that. > > > Unfortunately, regarding absolute paths, I had also tried to use cygwin > paths (`/cygdrive/c/...`) but that does not work either. > > > Hmm, that should work I think. Satish, do we have a machine to login and > test something like this? > > Thanks, > > Matt > > > Best, > Milan > > ??????? Original Message ??????? > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley > a ?crit : > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > Dear PETSc team, > > The way some options are passed to PETSc can be an issue on Windows, in > particular regarding the "log_view" option. > Since the colon ':' character is used as a separator, I do not see how > absolute paths should be provided on Windows (since the drive name includes > a colon). Is there a trick / workaround to overcome this? > > > I thought we used Cygwin paths, but maybe one of the Windows people knows > better. > > > Besides, when I try to use the ascii_xml format, it crashes in file > xmllogevent.c, on line 751: > illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > (since nestedEvents is NULL). > > > This likely means that you have unbalanced events. I can run this on > examples. For instance does SNES ex5 work for you? > > Thanks, > > Matt > > > I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > > Thanks for your help, > Best regards, > Milan Pelletier > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Mon Nov 22 12:04:43 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 13:04:43 -0500 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: On Mon, Nov 22, 2021 at 12:41 PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > @Matthew Knepley Yes, it works with main build! > That is good. I think we might need to push up the next point release then, since we have a Hypre bug for which a fix is non-obvious. Thanks, Matt > @Mark Adams I have attached the log_view output of ex4 > for your reference. But pcapply or pcsetup did not record any gpu flops. > > If it is not too much trouble can you send me the log_view output of > ksp/ksp/tutorial/ex45 using hypre? > > > > Many thanks! > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Monday, 22 November 2021 at 15:26 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , " > petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > On Mon, Nov 22, 2021 at 10:20 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thanks Stefano, > > > > I just did a clean build with no problems. Can you try a clean build of > main? > > > > Thanks, > > > > Matt > > > > > > I have another build without the --download-hypre-commit=origin/hypre_petsc > but that gives a different error. > > > > [kchockalingam at glados tutorials]$ ./ex4 -ksp_view -ksp_type cg -mat_type > hypre -pc_type hypre > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Error in external library > > [0]PETSC ERROR: Error in HYPRE_IJMatrixAssemble(): error code 12 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.15.3, Aug 06, 2021 > > [0]PETSC ERROR: ./ex4 on a named glados.dl.ac.uk by kchockalingam Mon > Nov 22 15:07:46 2021 > > [0]PETSC ERROR: Configure options > --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs > --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer" > FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 > -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 > --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ > --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1 > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --with-debugging=no > PETSC_ARCH=arch-ci-linux-cuda11-double > > [0]PETSC ERROR: #1 MatAssemblyEnd_HYPRE() at > /home/kchockalingam/tools/petsc-3.15.3/src/mat/impls/hypre/mhypre.c:1212 > > [0]PETSC ERROR: #2 MatAssemblyEnd() at > /home/kchockalingam/tools/petsc-3.15.3/src/mat/interface/matrix.c:5652 > > [0]PETSC ERROR: #3 main() at ex4.c:84 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -ksp_type cg > > [0]PETSC ERROR: -ksp_view > > [0]PETSC ERROR: -mat_type hypre > > [0]PETSC ERROR: -pc_type hypre > > *[0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov----------* > > -------------------------------------------------------------------------- > > > > Best, > > Karthik. > > > > *From: *Stefano Zampini > *Date: *Monday, 22 November 2021 at 14:46 > *To: *Matthew Knepley > *Cc: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk>, "petsc-users at mcs.anl.gov" < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] hypre on gpus > > > > You don't need to specify the HYPRE commit. Remove --download-hypre-commit=origin/hypre_petsc > from the configuration options > > > > Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley < > knepley at gmail.com> ha scritto: > > On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > Below is the entire error message: > > > > I cannot reproduce this: > > > > main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view > -ksp_type cg -mat_type hypre -pc_type hypre > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: hypre > rows=56, cols=56 > Norm of error 8.69801e-05 iterations 2 > > This is on the 'main' branch. So either there is some bug in release, or > something is strange on your end. Since we run Hypre tests for the CI, > > I am leaning toward the latter. Can you try the 'main' branch? We will > have to use this anyway if we want any fixes. > > > > Thanks, > > > > Matt > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 > > [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 > 11:33:41 2021 > > [0]PETSC ERROR: Configure options > --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 > --with-debugging=yes > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 > --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx > -with-fc=mpif90 > > [0]PETSC ERROR: #1 MatGetOwnershipRange() at > /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 > > [0]PETSC ERROR: #2 main() at ex4.c:40 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -ksp_type cg > > [0]PETSC ERROR: -ksp_view > > [0]PETSC ERROR: -mat_type hypre > > [0]PETSC ERROR: -pc_type hypre > > [0]PETSC ERROR: -use_gpu_aware_mpi 0 > > *[0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov----------* > > -------------------------------------------------------------------------- > > > > I have also attached the make.log. Thank you for having a look. > > > > Best, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Monday, 22 November 2021 at 13:41 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Mark Adams , "petsc-users at mcs.anl.gov" < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] hypre on gpus > > > > On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thank you for your response. I tried to run the same example > > > > petsc/src/ksp/ksp/tutorials$ *./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > but it crashed with the below error > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > > > Hi Karthik, > > > > Please do not clip the error message. We need the entire output. This > seems strange that you would get a logic error, > > since that should be the same on any architecture. So could you also send > the make.log? > > > > Thanks, > > > > Matt > > > > Below are the options used to configure hypre with cuda support. Do you > spot any mistakes? > > > > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > > > --with-cuda=1 > > --with-cuda-arch=70 > > --download-hypre=yes > > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > > --download-hypre-commit=origin/hypre_petsc > > --with-shared-libraries=1 > > --known-mpi-shared-libraries=1 > > --with-cc=mpicc > > --with-cxx=mpicxx > > -with-fc=mpif90 > > > > Best, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 19 November 2021 at 16:31 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > You should run with -options_left to check that your options are being > used. It may be -mat_type hypre. > > > > I have tested this: > > > > petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > You can add -log_view and that will print performance data for each method > like KSPSolve. > > > > If you are configured with a GPU there will be some extra columns that > give the percent of flops on the GPU. > > > > In the past hypre has not registered flops with us, but I do get flops > from hypre now, and -ksp_view showed that it did indeed use hypre. > > I saw that the flops were 100% GPU in hypre. > > > > > > On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on > gpus > > > > ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre > -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg > -ksp_monitor -log_view > > > > I profiled the run using NSYS - attached you find all the relevant files. > > Looking at the profile I doubt if hypre is running on gpus. The cuda > kernels are barely active. > > I don?t see any cuda kernel relevant to solve. > > Is my assessment correct? How can I verify if hypre indeed is running on > gpus? > > > > Best, > > Karthik. > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:47 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > I think you would want to use 'cuda' vec_type, but I . > > You might ask Hypre how one verifies that the GPU is used. > > Mark > > > > On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Yes, I used it for both cpu and gpu. Is that not okay? > > > > For gpu: -dm_mat_type hypre -dm_vec_type mpicuda > > > > For cpu: -dm_mat_type hypre -dm_vec_type mpi > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:28 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Did you use -dm_mat_type hypre on the GPU case ? > > > > On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I tried a different exercise ran the same problem on two cpu cores and on > two gpu: > > > > On gpu > > > > PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 > 9.65e+01 0 > > > > and on cpu > > > > PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 > > > > timings again are close but gpu version did a reduction 6.0e+00 but the > cpu version did not 0.0e+00. > > I am not sure if that is any indication if hypre ran on gpus? > > > > Thanks, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 16:36 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > > > > > On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > The PCApply timing on > > > > gpu > > > > PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 > 0.00e+00 0 > > > > and cpu > > > > PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > > > > > > You don't have GPUs. probably. > > Use -dm_mat_type hypre. > > > > are close. It is hard for me tell if hypre on gpu is on or not. > > > > Best, > > Karthik. > > > > > > *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Date: *Friday, 8 October 2021 at 14:55 > *To: *Mark Adams > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Thanks Mark, I will try your recommendations. > > Should I also change -dm_vec_type to hypre currently I have it as mpicuda? > > > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 14:33 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Hypre does not record its flops with PETSc's timers. > > Configure with and without CUDA and see if the timings change in PCApply. > > Hypre does not dynamically switch between CUDA and CPU solves at > this time, but you want to use -dm_mat_type hypre. > > Mark > > > > On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have > attached the python configuration file and -log_view output from running > the below command options > > > > mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 > -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type > hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 > -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor > > > > The problem was solved and converged but from the output file I suspect > hypre is not running on gpus as PCApply and DMCreate does *not* record > any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are > running on gpus. > > > > Can you please let me know if I need to add any extra flag to the attached > arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? > > > > Thanks, > > Karthik. > > > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Nov 22 12:10:15 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 22 Nov 2021 13:10:15 -0500 Subject: [petsc-users] hypre on gpus In-Reply-To: References: <35051739-B062-4DF3-B4E2-2C1297453609@stfc.ac.uk> <39D1801E-DC81-4F1E-912C-DBD78BDD01DB@stfc.ac.uk> <6215BB2D-14CB-42A4-9A22-31AE98B2237C@stfc.ac.uk> <8FCCCA6C-232B-4C13-91ED-F294EAAA733E@stfc.ac.uk> <07C9C641-A3F8-4685-8E63-AC4891EA621B@stfc.ac.uk> Message-ID: Ok, I was wrong about hypre registering its flops. Our KSP is 100 % GPU, but there are very few flops recorded (ie, no hypre flops). No flops in PCApply. That implies that hypre is in fact being used. I don't know of a way to verify from log_view if hypre is running on the GPU. You could try -ksp_view to see if there is anything, but you can just look at the KSPSOlve time on decent sized problems with and without GPUs in hypre. Now, I am not sure what you should do to get a CPU hypre. You can configure a PETSc without cuda. That is the surest way to go, but simply removing -mat_type hypre (or change to -mat_type aij) might work and give you CPU hypre. I am not sure. Anyway you should see a difference in solve times with the CPU solves if in fact you are getting the GPU solver. On Mon, Nov 22, 2021 at 12:41 PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > @Matthew Knepley Yes, it works with main build! > > > > @Mark Adams I have attached the log_view output of ex4 > for your reference. But pcapply or pcsetup did not record any gpu flops. > > If it is not too much trouble can you send me the log_view output of > ksp/ksp/tutorial/ex45 using hypre? > > > > Many thanks! > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Monday, 22 November 2021 at 15:26 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Stefano Zampini , " > petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > On Mon, Nov 22, 2021 at 10:20 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thanks Stefano, > > > > I just did a clean build with no problems. Can you try a clean build of > main? > > > > Thanks, > > > > Matt > > > > > > I have another build without the --download-hypre-commit=origin/hypre_petsc > but that gives a different error. > > > > [kchockalingam at glados tutorials]$ ./ex4 -ksp_view -ksp_type cg -mat_type > hypre -pc_type hypre > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Error in external library > > [0]PETSC ERROR: Error in HYPRE_IJMatrixAssemble(): error code 12 > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.15.3, Aug 06, 2021 > > [0]PETSC ERROR: ./ex4 on a named glados.dl.ac.uk by kchockalingam Mon > Nov 22 15:07:46 2021 > > [0]PETSC ERROR: Configure options > --package-prefix-hash=/home/kchockalingam/petsc-hash-pkgs > --with-make-test-np=2 COPTFLAGS="-g -O3 -fno-omit-frame-pointer" > FOPTFLAGS="-g -O3 -fno-omit-frame-pointer" CXXOPTFLAGS="-g -O3 > -fno-omit-frame-pointer" --with-cuda=1 --with-cuda-arch=70 > --with-blaslapack=1 --with-cuda-dir=/apps/packages/cuda/10.1/ > --with-mpi-dir=/apps/packages/gcc/7.3.0/openmpi/3.1.2 --download-hypre=1 > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 --with-debugging=no > PETSC_ARCH=arch-ci-linux-cuda11-double > > [0]PETSC ERROR: #1 MatAssemblyEnd_HYPRE() at > /home/kchockalingam/tools/petsc-3.15.3/src/mat/impls/hypre/mhypre.c:1212 > > [0]PETSC ERROR: #2 MatAssemblyEnd() at > /home/kchockalingam/tools/petsc-3.15.3/src/mat/interface/matrix.c:5652 > > [0]PETSC ERROR: #3 main() at ex4.c:84 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -ksp_type cg > > [0]PETSC ERROR: -ksp_view > > [0]PETSC ERROR: -mat_type hypre > > [0]PETSC ERROR: -pc_type hypre > > *[0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov----------* > > -------------------------------------------------------------------------- > > > > Best, > > Karthik. > > > > *From: *Stefano Zampini > *Date: *Monday, 22 November 2021 at 14:46 > *To: *Matthew Knepley > *Cc: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk>, "petsc-users at mcs.anl.gov" < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] hypre on gpus > > > > You don't need to specify the HYPRE commit. Remove --download-hypre-commit=origin/hypre_petsc > from the configuration options > > > > Il giorno lun 22 nov 2021 alle ore 17:29 Matthew Knepley < > knepley at gmail.com> ha scritto: > > On Mon, Nov 22, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hi Matt, > > > > Below is the entire error message: > > > > I cannot reproduce this: > > > > main $:/PETSc3/petsc/petsc-dev/src/ksp/ksp/tutorials$ ./ex4 -ksp_view > -ksp_type cg -mat_type hypre -pc_type hypre > KSP Object: 1 MPI processes > type: cg > maximum iterations=10000, initial guess is zero > tolerances: relative=0.000138889, absolute=1e-50, divergence=10000. > left preconditioning > using PRECONDITIONED norm type for convergence test > PC Object: 1 MPI processes > type: hypre > HYPRE BoomerAMG preconditioning > Cycle type V > Maximum number of levels 25 > Maximum number of iterations PER hypre call 1 > Convergence tolerance PER hypre call 0. > Threshold for strong coupling 0.25 > Interpolation truncation factor 0. > Interpolation: max elements per row 0 > Number of levels of aggressive coarsening 0 > Number of paths for aggressive coarsening 1 > Maximum row sums 0.9 > Sweeps down 1 > Sweeps up 1 > Sweeps on coarse 1 > Relax down symmetric-SOR/Jacobi > Relax up symmetric-SOR/Jacobi > Relax on coarse Gaussian-elimination > Relax weight (all) 1. > Outer relax weight (all) 1. > Using CF-relaxation > Not using more complex smoothers. > Measure type local > Coarsen type Falgout > Interpolation type classical > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: hypre > rows=56, cols=56 > Norm of error 8.69801e-05 iterations 2 > > This is on the 'main' branch. So either there is some bug in release, or > something is strange on your end. Since we run Hypre tests for the CI, > > I am leaning toward the latter. Can you try the 'main' branch? We will > have to use this anyway if we want any fixes. > > > > Thanks, > > > > Matt > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > [0]PETSC ERROR: Petsc Release Version 3.16.0, Sep 29, 2021 > > [0]PETSC ERROR: ./ex4 on a named sqg2b4.bullx by kxc07-lxm25 Mon Nov 22 > 11:33:41 2021 > > [0]PETSC ERROR: Configure options > --prefix=/lustre/scafellpike/local/apps/gcc7/petsc/3.16.0-cuda11.2 > --with-debugging=yes > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > --with-cuda=1 --with-cuda-arch=70 --download-hypre=yes > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > --download-hypre-commit=origin/hypre_petsc --with-shared-libraries=1 > --known-mpi-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx > -with-fc=mpif90 > > [0]PETSC ERROR: #1 MatGetOwnershipRange() at > /netfs/smain01/scafellpike/local/package_build/build/rja87-build/petsc-cuda-3.16.0/src/mat/interface/matrix.c:6784 > > [0]PETSC ERROR: #2 main() at ex4.c:40 > > [0]PETSC ERROR: PETSc Option Table entries: > > [0]PETSC ERROR: -ksp_type cg > > [0]PETSC ERROR: -ksp_view > > [0]PETSC ERROR: -mat_type hypre > > [0]PETSC ERROR: -pc_type hypre > > [0]PETSC ERROR: -use_gpu_aware_mpi 0 > > *[0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov----------* > > -------------------------------------------------------------------------- > > > > I have also attached the make.log. Thank you for having a look. > > > > Best, > > Karthik. > > > > *From: *Matthew Knepley > *Date: *Monday, 22 November 2021 at 13:41 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *Mark Adams , "petsc-users at mcs.anl.gov" < > petsc-users at mcs.anl.gov> > *Subject: *Re: [petsc-users] hypre on gpus > > > > On Mon, Nov 22, 2021 at 6:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Thank you for your response. I tried to run the same example > > > > petsc/src/ksp/ksp/tutorials$ *./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > but it crashed with the below error > > > > *[0]PETSC ERROR: --------------------- Error Message > --------------------------------------------------------------* > > [0]PETSC ERROR: Object is in wrong state > > [0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the > matrix has not yet been factored on argument 1 "mat" before > MatGetOwnershipRange() > > > > Hi Karthik, > > > > Please do not clip the error message. We need the entire output. This > seems strange that you would get a logic error, > > since that should be the same on any architecture. So could you also send > the make.log? > > > > Thanks, > > > > Matt > > > > Below are the options used to configure hypre with cuda support. Do you > spot any mistakes? > > > > --with-blaslapack-dir=/lustre/scafellpike/local/apps/intel/intel_cs/2018.0.128/mkl > > > --with-cuda=1 > > --with-cuda-arch=70 > > --download-hypre=yes > > --download-hypre-configure-arguments=HYPRE_CUDA_SM=70 > > --download-hypre-commit=origin/hypre_petsc > > --with-shared-libraries=1 > > --known-mpi-shared-libraries=1 > > --with-cc=mpicc > > --with-cxx=mpicxx > > -with-fc=mpif90 > > > > Best, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 19 November 2021 at 16:31 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > You should run with -options_left to check that your options are being > used. It may be -mat_type hypre. > > > > I have tested this: > > > > petsc/src/ksp/ksp/tutorials$ *srun -n2 ./ex4 -ksp_type cg -mat_type hypre > -ksp_view -pc_type hypre* > > > > You can add -log_view and that will print performance data for each method > like KSPSolve. > > > > If you are configured with a GPU there will be some extra columns that > give the percent of flops on the GPU. > > > > In the past hypre has not registered flops with us, but I do get flops > from hypre now, and -ksp_view showed that it did indeed use hypre. > > I saw that the flops were 100% GPU in hypre. > > > > > > On Fri, Nov 19, 2021 at 10:47 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on > gpus > > > > ./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre > -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg > -ksp_monitor -log_view > > > > I profiled the run using NSYS - attached you find all the relevant files. > > Looking at the profile I doubt if hypre is running on gpus. The cuda > kernels are barely active. > > I don?t see any cuda kernel relevant to solve. > > Is my assessment correct? How can I verify if hypre indeed is running on > gpus? > > > > Best, > > Karthik. > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:47 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > I think you would want to use 'cuda' vec_type, but I . > > You might ask Hypre how one verifies that the GPU is used. > > Mark > > > > On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Yes, I used it for both cpu and gpu. Is that not okay? > > > > For gpu: -dm_mat_type hypre -dm_vec_type mpicuda > > > > For cpu: -dm_mat_type hypre -dm_vec_type mpi > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 18:28 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Did you use -dm_mat_type hypre on the GPU case ? > > > > On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > I tried a different exercise ran the same problem on two cpu cores and on > two gpu: > > > > On gpu > > > > PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 > 9.65e+01 0 > > > > and on cpu > > > > PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 16 0 0 0 0 16 0 0 0 0 0 > > > > timings again are close but gpu version did a reduction 6.0e+00 but the > cpu version did not 0.0e+00. > > I am not sure if that is any indication if hypre ran on gpus? > > > > Thanks, > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 16:36 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > > > > > On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > The PCApply timing on > > > > gpu > > > > PCApply 6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 39 0 0 0 0 39 0 0 0 0 0 0 0 0.00e+00 0 > 0.00e+00 0 > > > > and cpu > > > > PCApply 6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 41 0 0 0 0 41 0 0 0 0 0 > > > > > > You don't have GPUs. probably. > > Use -dm_mat_type hypre. > > > > are close. It is hard for me tell if hypre on gpu is on or not. > > > > Best, > > Karthik. > > > > > > *From: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Date: *Friday, 8 October 2021 at 14:55 > *To: *Mark Adams > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Thanks Mark, I will try your recommendations. > > Should I also change -dm_vec_type to hypre currently I have it as mpicuda? > > > > Karthik. > > > > > > *From: *Mark Adams > *Date: *Friday, 8 October 2021 at 14:33 > *To: *"Chockalingam, Karthikeyan (STFC,DL,HC)" < > karthikeyan.chockalingam at stfc.ac.uk> > *Cc: *"petsc-users at mcs.anl.gov" > *Subject: *Re: [petsc-users] hypre on gpus > > > > Hypre does not record its flops with PETSc's timers. > > Configure with and without CUDA and see if the timings change in PCApply. > > Hypre does not dynamically switch between CUDA and CPU solves at > this time, but you want to use -dm_mat_type hypre. > > Mark > > > > On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > > Hello, > > > > I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have > attached the python configuration file and -log_view output from running > the below command options > > > > mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 > -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type gmres -pc_type > hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 > -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor > > > > The problem was solved and converged but from the output file I suspect > hypre is not running on gpus as PCApply and DMCreate does *not* record > any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are > running on gpus. > > > > Can you please let me know if I need to add any extra flag to the attached > arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus? > > > > Thanks, > > Karthik. > > > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > -- > > Stefano > > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Nov 22 12:31:00 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 22 Nov 2021 13:31:00 -0500 Subject: [petsc-users] dmplex extruded layers In-Reply-To: References: Message-ID: This is a Matt question but if ex10 does what you want, and works, then I would copy the construction sequence carefully. You are probably doing something slightly different. * I don't think DMPlexFindVertices will work for you. It looks like you need the exact points that you are looking for. I have a branch (adams/snes-ex56-update) that creates a label for simple vertical faces that looks at the coordinates of a face and sees if they all have the same x or y coordinate. You might want to modify that to look for all vertices with the same radius. * I visualize subdomains by giving each cell one dof and putting the processor ID in that dof. You get a global view but the subdomains are different colors. Mark On Mon, Nov 22, 2021 at 12:10 PM Bhargav Subramanya < bhargav.subramanya at kaust.edu.sa> wrote: > Dear All, > > I have a prismatic mesh generated by extruding a base icosahedron > spherical surface mesh. The mesh is distributed on 2 processes after > extrusion. The dof for my problem are defined only on the simplices of the > spherical layers. > > 1. For a given spherical layer, I want to get points, which are basically > simplices, lying on that layer only. DMPlexGetHeightStratum returns the > points on all the spherical Layers. I can probably use DMPlexFindVertices > (based on the radius of the spherical layer) and > DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to > retrieve the points (which are simplices) on a given spherical layer in the > extruded mesh? > > 2. I am able to visualize the entire mesh using dm_view. Is there a way to > get the mesh file for the local dm from a specific process? > > 3. One of my previous emails might have got lost as I mistakenly attached > a large mesh file and sent it. So I am repeating the question here. > DMPlexExtrude gives the following error after distributing the base 2D > spherical mesh. Both refinement or/and extrusion after dm distribution does > not work for me. In fact, I tried with src/dm/impls/plex/tutorials/ex10.c > also. Although the mesh is distributed after the extrusion in ex10.c (which > is working fine for me), I tried to distribute before extrusion, and I get > the following error. Could you please suggest where I might be making a > mistake? > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: This DMPlex is distributed but its PointSF has no graph set > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.16.1, unknown > [0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named kl-21859 > by subrambm Mon Nov 22 19:47:14 2021 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > --download-superlu_dist --download-hypre --download-fiat > --download-generator --download-triangle --download-tetgen --download-chaco > --download-make -download-boost --download-cmake --download-ml > --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git > --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch > --download-hdf5 --force > [0]PETSC ERROR: #1 DMPlexCheckPointSF() at > /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554 > [0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at > /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595 > [0]PETSC ERROR: #3 DMPlexInterpolate() at > /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357 > [0]PETSC ERROR: #4 DMPlexExtrude() at > /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543 > [0]PETSC ERROR: #5 CreateMesh() at ex10.c:161 > [0]PETSC ERROR: #6 main() at ex10.c:180 > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -dm_plex_extrude_layers 3 > [0]PETSC ERROR: -dm_view vtk:mesh.vtk > [0]PETSC ERROR: -init_dm_plex_dim 2 > [0]PETSC ERROR: -petscpartitioner_type simple > [0]PETSC ERROR: -srf_dm_distribute > [0]PETSC ERROR: -srf_dm_refine 0 > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > > Thanks, > Bhargav > > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From balay at mcs.anl.gov Mon Nov 22 12:31:32 2021 From: balay at mcs.anl.gov (Satish Balay) Date: Mon, 22 Nov 2021 12:31:32 -0600 (CST) Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> Message-ID: Ah yes.. PATH=C:\cygwin64\usr\local\bin;C:\cygwin64\bin;C:\cygwin64\bin;C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2020.2.254\windows\mpi\intel64\bin;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64_win\mpirt;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32_win\mpirt;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64_win\compiler;C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32_win\compiler;C:\Program Files\Microsoft MPI\Bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\cygwin64\lib\lapack Satish On Mon, 22 Nov 2021, Matthew Knepley wrote: > On Mon, Nov 22, 2021 at 11:57 AM Barry Smith wrote: > > > > > What does Windows use for a path separator? > > > > semicolon > > Maybe we could use either one as a separator, since semicolon is also used > by the UNIX shell. > > Matt > > > > On Nov 22, 2021, at 11:19 AM, Satish Balay via petsc-users < > > petsc-users at mcs.anl.gov> wrote: > > > > Well our build tools use cygwin - but the library is built with MS/Intel > > compilers - they use paths in windows notation. > > > > I'm not sure if there is a way to get windows PATHs without ':' > > > > Satish > > > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > > > On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < > > milan.pelletier at protonmail.com> wrote: > > > > Matt, > > Thanks for the answer. I could fix my issue regarding unbalanced events - > > and so SNES ex5 does work well too. > > By the way, such imbalance does not trigger errors when log_view is using > > plain ascii output, but it does fail when ascii_xml is used instead. > > > > > > Yes, we have a "failsafe" for the regular logging, but the XML is > > specifically hierarchical and we do not know how to close it. We could take > > another look at that. > > > > > > Unfortunately, regarding absolute paths, I had also tried to use cygwin > > paths (`/cygdrive/c/...`) but that does not work either. > > > > > > Hmm, that should work I think. Satish, do we have a machine to login and > > test something like this? > > > > Thanks, > > > > Matt > > > > > > Best, > > Milan > > > > ??????? Original Message ??????? > > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley > > a ?crit : > > > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > > petsc-users at mcs.anl.gov> wrote: > > > > Dear PETSc team, > > > > The way some options are passed to PETSc can be an issue on Windows, in > > particular regarding the "log_view" option. > > Since the colon ':' character is used as a separator, I do not see how > > absolute paths should be provided on Windows (since the drive name includes > > a colon). Is there a trick / workaround to overcome this? > > > > > > I thought we used Cygwin paths, but maybe one of the Windows people knows > > better. > > > > > > Besides, when I try to use the ascii_xml format, it crashes in file > > xmllogevent.c, on line 751: > > illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > > (since nestedEvents is NULL). > > > > > > This likely means that you have unbalanced events. I can run this on > > examples. For instance does SNES ex5 work for you? > > > > Thanks, > > > > Matt > > > > > > I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > > > > Thanks for your help, > > Best regards, > > Milan Pelletier > > > > > > > > -- > > What most experimenters take for granted before they begin their > > experiments is infinitely more interesting than any results to which their > > experiments lead. > > -- Norbert Wiener > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > From knepley at gmail.com Mon Nov 22 12:38:08 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 13:38:08 -0500 Subject: [petsc-users] dmplex extruded layers In-Reply-To: References: Message-ID: On Mon, Nov 22, 2021 at 12:10 PM Bhargav Subramanya < bhargav.subramanya at kaust.edu.sa> wrote: > Dear All, > > I have a prismatic mesh generated by extruding a base icosahedron > spherical surface mesh. The mesh is distributed on 2 processes after > extrusion. The dof for my problem are defined only on the simplices of the > spherical layers. > > 1. For a given spherical layer, I want to get points, which are basically > simplices, lying on that layer only. DMPlexGetHeightStratum returns the > points on all the spherical Layers. I can probably use DMPlexFindVertices > (based on the radius of the spherical layer) and > DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to > retrieve the points (which are simplices) on a given spherical layer in the > extruded mesh? > DMPlexGetHeightStratum() refers to height in the Hasse Diagram, which is a DAG, not in the mesh. For example, height 0 points are the cells, height 1 are the faces, etc. I believe the default numbering for extrusion (in the main branch), is that all vertices produced from a given vertex be in order. This would mean that if v were the vertex point number, then (v - vStart) % Nl == l where Nl is the number of layers and l is the layer of that vertex. It is also the same for triangles. So if you want to segregate each shell, after extrusion, make a label that gives triangles this marker, and then use DMPlexLabelComplete(). Then after refinement you will still have the shells labeled correctly. I would be happy to help you make an example that does this. It seems cool. > 2. I am able to visualize the entire mesh using dm_view. Is there a way to > get the mesh file for the local dm from a specific process? > You can use -dm_partition_view which outputs a field with the process number. Then use Clip inside Paraview and clip to the process number you want, or just view that field so each process has a different color. > 3. One of my previous emails might have got lost as I mistakenly attached > a large mesh file and sent it. So I am repeating the question here. > DMPlexExtrude gives the following error after distributing the base 2D > spherical mesh. Both refinement or/and extrusion after dm distribution does > not work for me. In fact, I tried with src/dm/impls/plex/tutorials/ex10.c > also. Although the mesh is distributed after the extrusion in ex10.c (which > is working fine for me), I tried to distribute before extrusion, and I get > the following error. Could you please suggest where I might be making a > mistake? > So you want to distribute the mesh before extruding. For that small example (using the main branch), I run mpiexec -n 3 ./meshtest -dm_plex_shape sphere -dm_refine_pre 2 -dm_distribute -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3 and I get the attached picture. Thanks, Matt > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Object is in wrong state > [0]PETSC ERROR: This DMPlex is distributed but its PointSF has no graph set > [0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > [0]PETSC ERROR: Petsc Release Version 3.16.1, unknown > [0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named kl-21859 > by subrambm Mon Nov 22 19:47:14 2021 > [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ > --with-fc=gfortran --download-mpich --download-fblaslapack > --download-superlu_dist --download-hypre --download-fiat > --download-generator --download-triangle --download-tetgen --download-chaco > --download-make -download-boost --download-cmake --download-ml > --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git > --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch > --download-hdf5 --force > [0]PETSC ERROR: #1 DMPlexCheckPointSF() at > /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554 > [0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at > /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595 > [0]PETSC ERROR: #3 DMPlexInterpolate() at > /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357 > [0]PETSC ERROR: #4 DMPlexExtrude() at > /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543 > [0]PETSC ERROR: #5 CreateMesh() at ex10.c:161 > [0]PETSC ERROR: #6 main() at ex10.c:180 > [0]PETSC ERROR: PETSc Option Table entries: > [0]PETSC ERROR: -dm_plex_extrude_layers 3 > [0]PETSC ERROR: -dm_view vtk:mesh.vtk > [0]PETSC ERROR: -init_dm_plex_dim 2 > [0]PETSC ERROR: -petscpartitioner_type simple > [0]PETSC ERROR: -srf_dm_distribute > [0]PETSC ERROR: -srf_dm_refine 0 > [0]PETSC ERROR: ----------------End of Error Message -------send entire > error message to petsc-maint at mcs.anl.gov---------- > > Thanks, > Bhargav > > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: annulus.png Type: image/png Size: 145625 bytes Desc: not available URL: From knepley at gmail.com Mon Nov 22 12:44:04 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 22 Nov 2021 13:44:04 -0500 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> Message-ID: On Mon, Nov 22, 2021 at 1:31 PM Satish Balay wrote: > Ah yes.. > > PATH=C:\cygwin64\usr\local\bin;C:\cygwin64\bin;C:\cygwin64\bin;C:\Program > Files > (x86)\IntelSWTools\compilers_and_libraries_2020.2.254\windows\mpi\intel64\bin;C:\Program > Files (x86)\Common Files\Intel\Shared > Libraries\redist\intel64_win\mpirt;C:\Program Files (x86)\Common > Files\Intel\Shared Libraries\redist\ia32_win\mpirt;C:\Program Files > (x86)\Common Files\Intel\Shared > Libraries\redist\intel64_win\compiler;C:\Program Files (x86)\Common > Files\Intel\Shared Libraries\redist\ia32_win\compiler;C:\Program > Files\Microsoft > MPI\Bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\cygwin64\lib\lapack > The user would be responsible for escaping the semicolon if used in the shell unfortunately. Thanks, Matt > Satish > > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > > On Mon, Nov 22, 2021 at 11:57 AM Barry Smith wrote: > > > > > > > > What does Windows use for a path separator? > > > > > > > semicolon > > > > Maybe we could use either one as a separator, since semicolon is also > used > > by the UNIX shell. > > > > Matt > > > > > > > On Nov 22, 2021, at 11:19 AM, Satish Balay via petsc-users < > > > petsc-users at mcs.anl.gov> wrote: > > > > > > Well our build tools use cygwin - but the library is built with > MS/Intel > > > compilers - they use paths in windows notation. > > > > > > I'm not sure if there is a way to get windows PATHs without ':' > > > > > > Satish > > > > > > On Mon, 22 Nov 2021, Matthew Knepley wrote: > > > > > > On Mon, Nov 22, 2021 at 10:36 AM Milan Pelletier < > > > milan.pelletier at protonmail.com> wrote: > > > > > > Matt, > > > Thanks for the answer. I could fix my issue regarding unbalanced > events - > > > and so SNES ex5 does work well too. > > > By the way, such imbalance does not trigger errors when log_view is > using > > > plain ascii output, but it does fail when ascii_xml is used instead. > > > > > > > > > Yes, we have a "failsafe" for the regular logging, but the XML is > > > specifically hierarchical and we do not know how to close it. We could > take > > > another look at that. > > > > > > > > > Unfortunately, regarding absolute paths, I had also tried to use cygwin > > > paths (`/cygdrive/c/...`) but that does not work either. > > > > > > > > > Hmm, that should work I think. Satish, do we have a machine to login > and > > > test something like this? > > > > > > Thanks, > > > > > > Matt > > > > > > > > > Best, > > > Milan > > > > > > ??????? Original Message ??????? > > > Le lundi 22 novembre 2021 ? 2:39 PM, Matthew Knepley < > knepley at gmail.com> > > > a ?crit : > > > > > > On Mon, Nov 22, 2021 at 5:38 AM Milan Pelletier via petsc-users < > > > petsc-users at mcs.anl.gov> wrote: > > > > > > Dear PETSc team, > > > > > > The way some options are passed to PETSc can be an issue on Windows, in > > > particular regarding the "log_view" option. > > > Since the colon ':' character is used as a separator, I do not see how > > > absolute paths should be provided on Windows (since the drive name > includes > > > a colon). Is there a trick / workaround to overcome this? > > > > > > > > > I thought we used Cygwin paths, but maybe one of the Windows people > knows > > > better. > > > > > > > > > Besides, when I try to use the ascii_xml format, it crashes in file > > > xmllogevent.c, on line 751: > > > illegalEvent = 1+nestedEvents[nNestedEvents-1].nstEvent; > > > (since nestedEvents is NULL). > > > > > > > > > This likely means that you have unbalanced events. I can run this on > > > examples. For instance does SNES ex5 work for you? > > > > > > Thanks, > > > > > > Matt > > > > > > > > > I'm using PETSc version 3.16.1 (using `git checkout tags/v3.16.1`). > > > > > > Thanks for your help, > > > Best regards, > > > Milan Pelletier > > > > > > > > > > > > -- > > > What most experimenters take for granted before they begin their > > > experiments is infinitely more interesting than any results to which > their > > > experiments lead. > > > -- Norbert Wiener > > > > > > https://www.cse.buffalo.edu/~knepley/ > > > > > > > > > > > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Mon Nov 22 12:58:48 2021 From: bsmith at petsc.dev (Barry Smith) Date: Mon, 22 Nov 2021 13:58:48 -0500 Subject: [petsc-users] Issues with log_view on windows In-Reply-To: References: <09E464FC-81FE-4654-8D10-45369141B352@petsc.dev> Message-ID: I meant the separator in PATH, on Unix one has "/usr/bin:/usr/local/bin" is Windows "\something;\somethingelse"? Barry > On Nov 22, 2021, at 11:59 AM, Pilhwa Lee wrote: > > I weakly recall there is a difference of ?\? and ?/? between Mac and Windows. From kuang-chung.wang at intel.com Wed Nov 24 00:15:11 2021 From: kuang-chung.wang at intel.com (Wang, Kuang-chung) Date: Wed, 24 Nov 2021 06:15:11 +0000 Subject: [petsc-users] Orthogonality of eigenvectors in SLEPC Message-ID: Dear Jose : I came across this thread describing issue using krylovschur and finding eigenvectors non-orthogonal. https://lists.mcs.anl.gov/pipermail/petsc-users/2014-October/023360.html I furthermore have tested by reducing the tolerance as highlighted below from 1e-12 to 1e-16 with no luck. Could you please suggest options/sources to try out ? Thanks a lot for sharing your knowledge! Sincere, Kuang-Chung Wang ======================================================= Kuang-Chung Wang Computational and Modeling Technology Intel Corporation Hillsboro OR 97124 ======================================================= Here are more info: 1. slepc/3.7.4 2. output message from by doing EPSView(eps,PETSC_NULL): EPS Object: 1 MPI processes type: krylovschur Krylov-Schur: 50% of basis vectors kept after restart Krylov-Schur: using the locking variant problem type: non-hermitian eigenvalue problem selected portion of the spectrum: closest to target: 20.1161 (in magnitude) number of eigenvalues (nev): 40 number of column vectors (ncv): 81 maximum dimension of projected problem (mpd): 81 maximum number of iterations: 1000 tolerance: 1e-12 convergence test: relative to the eigenvalue BV Object: 1 MPI processes type: svec 82 columns of global length 2988 vector orthogonalization method: classical Gram-Schmidt orthogonalization refinement: always block orthogonalization method: Gram-Schmidt doing matmult as a single matrix-matrix product DS Object: 1 MPI processes type: nhep ST Object: 1 MPI processes type: sinvert shift: 20.1161 number of matrices: 1 KSP Object: (st_) 1 MPI processes type: preonly maximum iterations=1000, initial guess is zero tolerances: relative=1.12005e-09, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (st_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=2988, cols=2988 package used to perform factorization: mumps total: nonzeros=614160, allocated nonzeros=614160 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequential matrix ordering):7 ICNTL(8) (scaling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 0 ICNTL(19) (Schur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 0 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -24 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0. CNTL(4) (value of static pivoting): -1. CNTL(5) (fixation for null pivots): 0. RINFO(1) (local estimated flops for the elimination after analysis): [0] 8.15668e+07 RINFO(2) (local estimated flops for the assembly after factorization): [0] 892584. RINFO(3) (local estimated flops for the elimination after factorization): [0] 8.15668e+07 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 16 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 16 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 2988 RINFOG(1) (global estimated flops for the elimination after analysis): 8.15668e+07 RINFOG(2) (global estimated flops for the assembly after factorization): 892584. RINFOG(3) (global estimated flops for the elimination after factorization): 8.15668e+07 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 614160 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 31971 INFOG(5) (estimated maximum front size in the complete tree): 246 INFOG(6) (number of nodes in the complete tree): 197 INFOG(7) (ordering option effectively use after analysis): 2 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 614160 INFOG(10) (total integer space store the matrix factors after factorization): 31971 INFOG(11) (order of largest frontal matrix after factorization): 246 INFOG(12) (number of off-diagonal pivots): 0 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 16 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 16 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 16 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 16 INFOG(20) (estimated number of entries in the factors): 614160 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 14 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 14 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 614160 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 13, 13 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=2988, cols=2988 total: nonzeros=151488, allocated nonzeros=151488 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 996 nodes, limit used is 5 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jroman at dsic.upv.es Wed Nov 24 08:20:16 2021 From: jroman at dsic.upv.es (Jose E. Roman) Date: Wed, 24 Nov 2021 15:20:16 +0100 Subject: [petsc-users] Orthogonality of eigenvectors in SLEPC In-Reply-To: References: Message-ID: In Hermitian eigenproblems orthogonality of eigenvectors is guaranteed/enforced. But you are solving the problem as non-Hermitian. If your matrix is Hermitian, make sure you solve it as a HEP, and make sure that your matrix is numerically Hermitian. If your matrix is non-Hermitian, then you cannot expect the eigenvectors to be orthogonal. What you can do in this case is get an orthogonal basis of the computed eigenspace, see https://slepc.upv.es/documentation/current/docs/manualpages/EPS/EPSGetInvariantSubspace.html By the way, version 3.7 is more than 5 years old, it is better if you can upgrade to a more recent version. Jose > El 24 nov 2021, a las 7:15, Wang, Kuang-chung escribi?: > > Dear Jose : > I came across this thread describing issue using krylovschur and finding eigenvectors non-orthogonal. > https://lists.mcs.anl.gov/pipermail/petsc-users/2014-October/023360.html > > I furthermore have tested by reducing the tolerance as highlighted below from 1e-12 to 1e-16 with no luck. > Could you please suggest options/sources to try out ? > Thanks a lot for sharing your knowledge! > > Sincere, > Kuang-Chung Wang > > ======================================================= > Kuang-Chung Wang > Computational and Modeling Technology > Intel Corporation > Hillsboro OR 97124 > ======================================================= > > Here are more info: > ? slepc/3.7.4 > ? output message from by doing EPSView(eps,PETSC_NULL): > EPS Object: 1 MPI processes > type: krylovschur > Krylov-Schur: 50% of basis vectors kept after restart > Krylov-Schur: using the locking variant > problem type: non-hermitian eigenvalue problem > selected portion of the spectrum: closest to target: 20.1161 (in magnitude) > number of eigenvalues (nev): 40 > number of column vectors (ncv): 81 > maximum dimension of projected problem (mpd): 81 > maximum number of iterations: 1000 > tolerance: 1e-12 > convergence test: relative to the eigenvalue > BV Object: 1 MPI processes > type: svec > 82 columns of global length 2988 > vector orthogonalization method: classical Gram-Schmidt > orthogonalization refinement: always > block orthogonalization method: Gram-Schmidt > doing matmult as a single matrix-matrix product > DS Object: 1 MPI processes > type: nhep > ST Object: 1 MPI processes > type: sinvert > shift: 20.1161 > number of matrices: 1 > KSP Object: (st_) 1 MPI processes > type: preonly > maximum iterations=1000, initial guess is zero > tolerances: relative=1.12005e-09, absolute=1e-50, divergence=10000. > left preconditioning > using NONE norm type for convergence test > PC Object: (st_) 1 MPI processes > type: lu > LU: out-of-place factorization > tolerance for zero pivot 2.22045e-14 > matrix ordering: nd > factor fill ratio given 0., needed 0. > Factored matrix follows: > Mat Object: 1 MPI processes > type: seqaij > rows=2988, cols=2988 > package used to perform factorization: mumps > total: nonzeros=614160, allocated nonzeros=614160 > total number of mallocs used during MatSetValues calls =0 > MUMPS run parameters: > SYM (matrix type): 0 > PAR (host participation): 1 > ICNTL(1) (output for error): 6 > ICNTL(2) (output of diagnostic msg): 0 > ICNTL(3) (output for global info): 0 > ICNTL(4) (level of printing): 0 > ICNTL(5) (input mat struct): 0 > ICNTL(6) (matrix prescaling): 7 > ICNTL(7) (sequential matrix ordering):7 > ICNTL(8) (scaling strategy): 77 > ICNTL(10) (max num of refinements): 0 > ICNTL(11) (error analysis): 0 > ICNTL(12) (efficiency control): 1 > ICNTL(13) (efficiency control): 0 > ICNTL(14) (percentage of estimated workspace increase): 20 > ICNTL(18) (input mat struct): 0 > ICNTL(19) (Schur complement info): 0 > ICNTL(20) (rhs sparse pattern): 0 > ICNTL(21) (solution struct): 0 > ICNTL(22) (in-core/out-of-core facility): 0 > ICNTL(23) (max size of memory can be allocated locally):0 > ICNTL(24) (detection of null pivot rows): 0 > ICNTL(25) (computation of a null space basis): 0 > ICNTL(26) (Schur options for rhs or solution): 0 > ICNTL(27) (experimental parameter): -24 > ICNTL(28) (use parallel or sequential ordering): 1 > ICNTL(29) (parallel ordering): 0 > ICNTL(30) (user-specified set of entries in inv(A)): 0 > ICNTL(31) (factors is discarded in the solve phase): 0 > ICNTL(33) (compute determinant): 0 > CNTL(1) (relative pivoting threshold): 0.01 > CNTL(2) (stopping criterion of refinement): 1.49012e-08 > CNTL(3) (absolute pivoting threshold): 0. > CNTL(4) (value of static pivoting): -1. > CNTL(5) (fixation for null pivots): 0. > RINFO(1) (local estimated flops for the elimination after analysis): > [0] 8.15668e+07 > RINFO(2) (local estimated flops for the assembly after factorization): > [0] 892584. > RINFO(3) (local estimated flops for the elimination after factorization): > [0] 8.15668e+07 > INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): > [0] 16 > INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): > [0] 16 > INFO(23) (num of pivots eliminated on this processor after factorization): > [0] 2988 > RINFOG(1) (global estimated flops for the elimination after analysis): 8.15668e+07 > RINFOG(2) (global estimated flops for the assembly after factorization): 892584. > RINFOG(3) (global estimated flops for the elimination after factorization): 8.15668e+07 > (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) > INFOG(3) (estimated real workspace for factors on all processors after analysis): 614160 > INFOG(4) (estimated integer workspace for factors on all processors after analysis): 31971 > INFOG(5) (estimated maximum front size in the complete tree): 246 > INFOG(6) (number of nodes in the complete tree): 197 > INFOG(7) (ordering option effectively use after analysis): 2 > INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 > INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 614160 > INFOG(10) (total integer space store the matrix factors after factorization): 31971 > INFOG(11) (order of largest frontal matrix after factorization): 246 > INFOG(12) (number of off-diagonal pivots): 0 > INFOG(13) (number of delayed pivots after factorization): 0 > INFOG(14) (number of memory compress after factorization): 0 > INFOG(15) (number of steps of iterative refinement after solution): 0 > INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 16 > INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 16 > INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 16 > INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 16 > INFOG(20) (estimated number of entries in the factors): 614160 > INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 14 > INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 14 > INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 > INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 > INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 > INFOG(28) (after factorization: number of null pivots encountered): 0 > INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 614160 > INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 13, 13 > INFOG(32) (after analysis: type of analysis done): 1 > INFOG(33) (value used for ICNTL(8)): 7 > INFOG(34) (exponent of the determinant if determinant is requested): 0 > linear system matrix = precond matrix: > Mat Object: 1 MPI processes > type: seqaij > rows=2988, cols=2988 > total: nonzeros=151488, allocated nonzeros=151488 > total number of mallocs used during MatSetValues calls =0 > using I-node routines: found 996 nodes, limit used is 5 From bantingl at myumanitoba.ca Wed Nov 24 09:50:15 2021 From: bantingl at myumanitoba.ca (Lucas Banting) Date: Wed, 24 Nov 2021 15:50:15 +0000 Subject: [petsc-users] Orthogonality of eigenvectors in SLEPC In-Reply-To: References: Message-ID: Hi Kuang-Chung Wang , It says your eigenvalue problem type is: problem type: non-hermitian eigenvalue problem While the email chain you refer to is for a hermitian eigenvalue problem. Try changing your eigenvalue problem type to a hermitian or generalized hermitian problem and it should produce orthogonal eigenvectors. Lucas ________________________________ From: petsc-users on behalf of Wang, Kuang-chung Sent: Wednesday, November 24, 2021 12:15 AM To: petsc-users at mcs.anl.gov Cc: Obradovic, Borna ; Cea, Stephen M Subject: Re: [petsc-users] Orthogonality of eigenvectors in SLEPC Dear Jose : I came across this thread describing issue using krylovschur and finding eigenvectors non-orthogonal. https://lists.mcs.anl.gov/pipermail/petsc-users/2014-October/023360.html I furthermore have tested by reducing the tolerance as highlighted below from 1e-12 to 1e-16 with no luck. Could you please suggest options/sources to try out ? Thanks a lot for sharing your knowledge! Sincere, Kuang-Chung Wang ======================================================= Kuang-Chung Wang Computational and Modeling Technology Intel Corporation Hillsboro OR 97124 ======================================================= Here are more info: 1. slepc/3.7.4 2. output message from by doing EPSView(eps,PETSC_NULL): EPS Object: 1 MPI processes type: krylovschur Krylov-Schur: 50% of basis vectors kept after restart Krylov-Schur: using the locking variant problem type: non-hermitian eigenvalue problem selected portion of the spectrum: closest to target: 20.1161 (in magnitude) number of eigenvalues (nev): 40 number of column vectors (ncv): 81 maximum dimension of projected problem (mpd): 81 maximum number of iterations: 1000 tolerance: 1e-12 convergence test: relative to the eigenvalue BV Object: 1 MPI processes type: svec 82 columns of global length 2988 vector orthogonalization method: classical Gram-Schmidt orthogonalization refinement: always block orthogonalization method: Gram-Schmidt doing matmult as a single matrix-matrix product DS Object: 1 MPI processes type: nhep ST Object: 1 MPI processes type: sinvert shift: 20.1161 number of matrices: 1 KSP Object: (st_) 1 MPI processes type: preonly maximum iterations=1000, initial guess is zero tolerances: relative=1.12005e-09, absolute=1e-50, divergence=10000. left preconditioning using NONE norm type for convergence test PC Object: (st_) 1 MPI processes type: lu LU: out-of-place factorization tolerance for zero pivot 2.22045e-14 matrix ordering: nd factor fill ratio given 0., needed 0. Factored matrix follows: Mat Object: 1 MPI processes type: seqaij rows=2988, cols=2988 package used to perform factorization: mumps total: nonzeros=614160, allocated nonzeros=614160 total number of mallocs used during MatSetValues calls =0 MUMPS run parameters: SYM (matrix type): 0 PAR (host participation): 1 ICNTL(1) (output for error): 6 ICNTL(2) (output of diagnostic msg): 0 ICNTL(3) (output for global info): 0 ICNTL(4) (level of printing): 0 ICNTL(5) (input mat struct): 0 ICNTL(6) (matrix prescaling): 7 ICNTL(7) (sequential matrix ordering):7 ICNTL(8) (scaling strategy): 77 ICNTL(10) (max num of refinements): 0 ICNTL(11) (error analysis): 0 ICNTL(12) (efficiency control): 1 ICNTL(13) (efficiency control): 0 ICNTL(14) (percentage of estimated workspace increase): 20 ICNTL(18) (input mat struct): 0 ICNTL(19) (Schur complement info): 0 ICNTL(20) (rhs sparse pattern): 0 ICNTL(21) (solution struct): 0 ICNTL(22) (in-core/out-of-core facility): 0 ICNTL(23) (max size of memory can be allocated locally):0 ICNTL(24) (detection of null pivot rows): 0 ICNTL(25) (computation of a null space basis): 0 ICNTL(26) (Schur options for rhs or solution): 0 ICNTL(27) (experimental parameter): -24 ICNTL(28) (use parallel or sequential ordering): 1 ICNTL(29) (parallel ordering): 0 ICNTL(30) (user-specified set of entries in inv(A)): 0 ICNTL(31) (factors is discarded in the solve phase): 0 ICNTL(33) (compute determinant): 0 CNTL(1) (relative pivoting threshold): 0.01 CNTL(2) (stopping criterion of refinement): 1.49012e-08 CNTL(3) (absolute pivoting threshold): 0. CNTL(4) (value of static pivoting): -1. CNTL(5) (fixation for null pivots): 0. RINFO(1) (local estimated flops for the elimination after analysis): [0] 8.15668e+07 RINFO(2) (local estimated flops for the assembly after factorization): [0] 892584. RINFO(3) (local estimated flops for the elimination after factorization): [0] 8.15668e+07 INFO(15) (estimated size of (in MB) MUMPS internal data for running numerical factorization): [0] 16 INFO(16) (size of (in MB) MUMPS internal data used during numerical factorization): [0] 16 INFO(23) (num of pivots eliminated on this processor after factorization): [0] 2988 RINFOG(1) (global estimated flops for the elimination after analysis): 8.15668e+07 RINFOG(2) (global estimated flops for the assembly after factorization): 892584. RINFOG(3) (global estimated flops for the elimination after factorization): 8.15668e+07 (RINFOG(12) RINFOG(13))*2^INFOG(34) (determinant): (0.,0.)*(2^0) INFOG(3) (estimated real workspace for factors on all processors after analysis): 614160 INFOG(4) (estimated integer workspace for factors on all processors after analysis): 31971 INFOG(5) (estimated maximum front size in the complete tree): 246 INFOG(6) (number of nodes in the complete tree): 197 INFOG(7) (ordering option effectively use after analysis): 2 INFOG(8) (structural symmetry in percent of the permuted matrix after analysis): 100 INFOG(9) (total real/complex workspace to store the matrix factors after factorization): 614160 INFOG(10) (total integer space store the matrix factors after factorization): 31971 INFOG(11) (order of largest frontal matrix after factorization): 246 INFOG(12) (number of off-diagonal pivots): 0 INFOG(13) (number of delayed pivots after factorization): 0 INFOG(14) (number of memory compress after factorization): 0 INFOG(15) (number of steps of iterative refinement after solution): 0 INFOG(16) (estimated size (in MB) of all MUMPS internal data for factorization after analysis: value on the most memory consuming processor): 16 INFOG(17) (estimated size of all MUMPS internal data for factorization after analysis: sum over all processors): 16 INFOG(18) (size of all MUMPS internal data allocated during factorization: value on the most memory consuming processor): 16 INFOG(19) (size of all MUMPS internal data allocated during factorization: sum over all processors): 16 INFOG(20) (estimated number of entries in the factors): 614160 INFOG(21) (size in MB of memory effectively used during factorization - value on the most memory consuming processor): 14 INFOG(22) (size in MB of memory effectively used during factorization - sum over all processors): 14 INFOG(23) (after analysis: value of ICNTL(6) effectively used): 0 INFOG(24) (after analysis: value of ICNTL(12) effectively used): 1 INFOG(25) (after factorization: number of pivots modified by static pivoting): 0 INFOG(28) (after factorization: number of null pivots encountered): 0 INFOG(29) (after factorization: effective number of entries in the factors (sum over all processors)): 614160 INFOG(30, 31) (after solution: size in Mbytes of memory used during solution phase): 13, 13 INFOG(32) (after analysis: type of analysis done): 1 INFOG(33) (value used for ICNTL(8)): 7 INFOG(34) (exponent of the determinant if determinant is requested): 0 linear system matrix = precond matrix: Mat Object: 1 MPI processes type: seqaij rows=2988, cols=2988 total: nonzeros=151488, allocated nonzeros=151488 total number of mallocs used during MatSetValues calls =0 using I-node routines: found 996 nodes, limit used is 5 -------------- next part -------------- An HTML attachment was scrubbed... URL: From karthikeyan.chockalingam at stfc.ac.uk Wed Nov 24 11:26:34 2021 From: karthikeyan.chockalingam at stfc.ac.uk (Karthikeyan Chockalingam - STFC UKRI) Date: Wed, 24 Nov 2021 17:26:34 +0000 Subject: [petsc-users] GAMG memory consumption Message-ID: <5B483CB3-9B0A-4B8C-B940-8CA8DD8AC135@stfc.ac.uk> Hello, I would like to understand why more memory is consumed by -pc_type gamg compared to -pc_type mg for the same problem size ksp/ksp/tutorial: ./ex45 -da_grid_x 368 -da_grid_x 368 -da_grid_x 368 -ksp_type cg -pc_type mg Maximum (over computational time) process memory: total 1.9399e+10 max 9.7000e+09 min 9.6992e+09 -pc_type gamg Maximum (over computational time) process memory: total 4.9671e+10 max 2.4836e+10 min 2.4835e+10 Am I right in understanding that the memory limiting factor is ?max 2.4836e+10? as it is the maximum memory used at any given time? I have attached the -log_view output of both the preconditioners. Best regards, Karthik. This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ksp_cg_pc_gamg_ex45_N368_gpu_2.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ksp_cg_pc_mg_ex45_N368_gpu_2.txt URL: From bhargav.subramanya at kaust.edu.sa Wed Nov 24 11:27:09 2021 From: bhargav.subramanya at kaust.edu.sa (Bhargav Subramanya) Date: Wed, 24 Nov 2021 20:27:09 +0300 Subject: [petsc-users] dmplex extruded layers In-Reply-To: References: Message-ID: Dear Matt and Mark, Thank you very much for your reply. Your inputs are very useful to me. On Mon, Nov 22, 2021 at 9:38 PM Matthew Knepley wrote: > On Mon, Nov 22, 2021 at 12:10 PM Bhargav Subramanya < > bhargav.subramanya at kaust.edu.sa> wrote: > >> Dear All, >> >> I have a prismatic mesh generated by extruding a base icosahedron >> spherical surface mesh. The mesh is distributed on 2 processes after >> extrusion. The dof for my problem are defined only on the simplices of the >> spherical layers. >> >> 1. For a given spherical layer, I want to get points, which are basically >> simplices, lying on that layer only. DMPlexGetHeightStratum returns the >> points on all the spherical Layers. I can probably use DMPlexFindVertices >> (based on the radius of the spherical layer) and >> DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to >> retrieve the points (which are simplices) on a given spherical layer in the >> extruded mesh? >> > > DMPlexGetHeightStratum() refers to height in the Hasse Diagram, which is a > DAG, not in the mesh. For example, height 0 points are the cells, height 1 > are the faces, etc. > > I believe the default numbering for extrusion (in the main branch), is > that all vertices produced from a given vertex be in order. This would mean > that if v were the vertex point number, then > > (v - vStart) % Nl == l > > where Nl is the number of layers and l is the layer of that vertex. It is > also the same for triangles. So if you want to segregate each shell, after > extrusion, make a label that gives triangles this marker, and then use > DMPlexLabelComplete(). Then after refinement you will still have the shells > labeled correctly. > > I would be happy to help you make an example that does this. It seems cool. > ----- Sure, I am interested to make an example of it. And, I really appreciate your help. > > >> 2. I am able to visualize the entire mesh using dm_view. Is there a way >> to get the mesh file for the local dm from a specific process? >> > > You can use -dm_partition_view which outputs a field with the process > number. Then use Clip inside Paraview and clip to the process number you > want, > or just view that field so each process has a different color. > > >> 3. One of my previous emails might have got lost as I mistakenly attached >> a large mesh file and sent it. So I am repeating the question here. >> DMPlexExtrude gives the following error after distributing the base 2D >> spherical mesh. Both refinement or/and extrusion after dm distribution does >> not work for me. In fact, I tried with src/dm/impls/plex/tutorials/ex10.c >> also. Although the mesh is distributed after the extrusion in ex10.c (which >> is working fine for me), I tried to distribute before extrusion, and I get >> the following error. Could you please suggest where I might be making a >> mistake? >> > > So you want to distribute the mesh before extruding. For that small > example (using the main branch), I run > > mpiexec -n 3 ./meshtest -dm_plex_shape sphere -dm_refine_pre 2 > -dm_distribute -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3 > ----- I pulled the code from the main branch and configured my petsc again. I am now finally able to reproduce the mesh that you have attached. I am able to do parallel mesh refinement, followed by extrusion, and use the following code to check that. However, I still have one problem. Attached are the extruded mesh files. 1. Mesh doesn't seem to be refined properly near interfaces between the processes. I then used -init_dm_distribute_overlap of 1 and that fixed the problem. I hope this is the way to do that. 2. The parallel mesh refinement doesn't seem to take care of the spherical geometry (shown in the attached mesh file). Could you please suggest how to fix it? command line options: -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3 ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, "init_");CHKERRQ(ierr); ierr = DMSetFromOptions(dm);CHKERRQ(ierr); ierr = DMPlexIsDistributed(dm, &distributed); ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); CHKERRQ(ierr); ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT); ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr); ierr = DMSetFromOptions(dm);CHKERRQ(ierr); ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); > and I get the attached picture. > > Thanks, > > Matt > > >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Object is in wrong state >> [0]PETSC ERROR: This DMPlex is distributed but its PointSF has no graph >> set >> [0]PETSC ERROR: See https://petsc.org/release/faq/ >> >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.16.1, unknown >> [0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named kl-21859 >> by subrambm Mon Nov 22 19:47:14 2021 >> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >> --with-fc=gfortran --download-mpich --download-fblaslapack >> --download-superlu_dist --download-hypre --download-fiat >> --download-generator --download-triangle --download-tetgen --download-chaco >> --download-make -download-boost --download-cmake --download-ml >> --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git >> >> --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch >> --download-hdf5 --force >> [0]PETSC ERROR: #1 DMPlexCheckPointSF() at >> /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554 >> [0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at >> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595 >> [0]PETSC ERROR: #3 DMPlexInterpolate() at >> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357 >> [0]PETSC ERROR: #4 DMPlexExtrude() at >> /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543 >> [0]PETSC ERROR: #5 CreateMesh() at ex10.c:161 >> [0]PETSC ERROR: #6 main() at ex10.c:180 >> [0]PETSC ERROR: PETSc Option Table entries: >> [0]PETSC ERROR: -dm_plex_extrude_layers 3 >> [0]PETSC ERROR: -dm_view vtk:mesh.vtk >> [0]PETSC ERROR: -init_dm_plex_dim 2 >> [0]PETSC ERROR: -petscpartitioner_type simple >> [0]PETSC ERROR: -srf_dm_distribute >> [0]PETSC ERROR: -srf_dm_refine 0 >> [0]PETSC ERROR: ----------------End of Error Message -------send entire >> error message to petsc-maint at mcs.anl.gov---------- >> >> Thanks, >> Bhargav >> >> >> ------------------------------ >> This message and its contents, including attachments are intended solely >> for the original recipient. If you are not the intended recipient or have >> received this message in error, please notify me immediately and delete >> this message from your computer system. Any unauthorized use or >> distribution is prohibited. Please consider the environment before printing >> this email. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mesh_overlap_0.png Type: image/png Size: 342980 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mesh_overlap_1.png Type: image/png Size: 331015 bytes Desc: not available URL: From knepley at gmail.com Wed Nov 24 11:59:36 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Nov 2021 12:59:36 -0500 Subject: [petsc-users] dmplex extruded layers In-Reply-To: References: Message-ID: On Wed, Nov 24, 2021 at 12:27 PM Bhargav Subramanya < bhargav.subramanya at kaust.edu.sa> wrote: > Dear Matt and Mark, > > Thank you very much for your reply. Your inputs are very useful to me. > > On Mon, Nov 22, 2021 at 9:38 PM Matthew Knepley wrote: > >> On Mon, Nov 22, 2021 at 12:10 PM Bhargav Subramanya < >> bhargav.subramanya at kaust.edu.sa> wrote: >> >>> Dear All, >>> >>> I have a prismatic mesh generated by extruding a base icosahedron >>> spherical surface mesh. The mesh is distributed on 2 processes after >>> extrusion. The dof for my problem are defined only on the simplices of the >>> spherical layers. >>> >>> 1. For a given spherical layer, I want to get points, which are >>> basically simplices, lying on that layer only. DMPlexGetHeightStratum >>> returns the points on all the spherical Layers. I can probably use >>> DMPlexFindVertices (based on the radius of the spherical layer) and >>> DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to >>> retrieve the points (which are simplices) on a given spherical layer in the >>> extruded mesh? >>> >> >> DMPlexGetHeightStratum() refers to height in the Hasse Diagram, which is >> a DAG, not in the mesh. For example, height 0 points are the cells, height >> 1 are the faces, etc. >> >> I believe the default numbering for extrusion (in the main branch), is >> that all vertices produced from a given vertex be in order. This would mean >> that if v were the vertex point number, then >> >> (v - vStart) % Nl == l >> >> where Nl is the number of layers and l is the layer of that vertex. It is >> also the same for triangles. So if you want to segregate each shell, after >> extrusion, make a label that gives triangles this marker, and then use >> DMPlexLabelComplete(). Then after refinement you will still have the shells >> labeled correctly. >> >> I would be happy to help you make an example that does this. It seems >> cool. >> > > ----- Sure, I am interested to make an example of it. And, I really > appreciate your help. > >> >> >>> 2. I am able to visualize the entire mesh using dm_view. Is there a way >>> to get the mesh file for the local dm from a specific process? >>> >> >> You can use -dm_partition_view which outputs a field with the process >> number. Then use Clip inside Paraview and clip to the process number you >> want, >> or just view that field so each process has a different color. >> >> >>> 3. One of my previous emails might have got lost as I mistakenly >>> attached a large mesh file and sent it. So I am repeating the question >>> here. DMPlexExtrude gives the following error after distributing the base >>> 2D spherical mesh. Both refinement or/and extrusion after dm distribution >>> does not work for me. In fact, I tried >>> with src/dm/impls/plex/tutorials/ex10.c also. Although the mesh is >>> distributed after the extrusion in ex10.c (which is working fine for me), I >>> tried to distribute before extrusion, and I get the following error. Could >>> you please suggest where I might be making a mistake? >>> >> >> So you want to distribute the mesh before extruding. For that small >> example (using the main branch), I run >> >> mpiexec -n 3 ./meshtest -dm_plex_shape sphere -dm_refine_pre 2 >> -dm_distribute -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3 >> > > ----- I pulled the code from the main branch and configured my petsc > again. I am now finally able to reproduce the mesh that you have attached. > I am able to do parallel mesh refinement, followed by extrusion, and use > the following code to check that. However, I still have one problem. > Attached are the extruded mesh files. > > 1. Mesh doesn't seem to be refined properly near interfaces between the > processes. I then used -init_dm_distribute_overlap of 1 and that fixed the > problem. I hope this is the way to do that. > Hmm, something is wrong here. We should track this down and fix it. I do not see this. Could you tell me how to reproduce the problem? > 2. The parallel mesh refinement doesn't seem to take care of the spherical > geometry (shown in the attached mesh file). Could you please suggest how to > fix it? > Okay, this is more subtle. Let me explain how things work inside. 1) When you refine, it inserts new cells/faces/edges/vertices. When we insert a new vertex, we ask what the coordinates should be. If no special information is there, we just average the surrounding vertices. 2) Then we allow a second step that processed all coordinates in the mesh after refinement For the sphere, we use 2) to make all points stick to the implicit surface (radius R). We cannot do this for the extruded mesh, so we turn it off. What makes the most sense to me is to do 1) Serial mesh refinement to get enough cells to distribute evenly 2) Mesh distribution 3) Mesh refinement 4) Extrusion This should preserve the geometry and scale well. Does that sound right? Thanks, Matt > command line options: > -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 > -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_partition_view > -dm_view hdf5:mesh.h5 -dm_extrude 3 > > ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); > ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); > ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, > "init_");CHKERRQ(ierr); > ierr = DMSetFromOptions(dm);CHKERRQ(ierr); > > ierr = DMPlexIsDistributed(dm, &distributed); > ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); > CHKERRQ(ierr); > ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT); > > ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr); > ierr = DMSetFromOptions(dm);CHKERRQ(ierr); > ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); > > >> and I get the attached picture. >> >> Thanks, >> >> Matt >> >> >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Object is in wrong state >>> [0]PETSC ERROR: This DMPlex is distributed but its PointSF has no graph >>> set >>> [0]PETSC ERROR: See https://petsc.org/release/faq/ >>> >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Release Version 3.16.1, unknown >>> [0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named >>> kl-21859 by subrambm Mon Nov 22 19:47:14 2021 >>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>> --with-fc=gfortran --download-mpich --download-fblaslapack >>> --download-superlu_dist --download-hypre --download-fiat >>> --download-generator --download-triangle --download-tetgen --download-chaco >>> --download-make -download-boost --download-cmake --download-ml >>> --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git >>> >>> --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch >>> --download-hdf5 --force >>> [0]PETSC ERROR: #1 DMPlexCheckPointSF() at >>> /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554 >>> [0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at >>> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595 >>> [0]PETSC ERROR: #3 DMPlexInterpolate() at >>> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357 >>> [0]PETSC ERROR: #4 DMPlexExtrude() at >>> /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543 >>> [0]PETSC ERROR: #5 CreateMesh() at ex10.c:161 >>> [0]PETSC ERROR: #6 main() at ex10.c:180 >>> [0]PETSC ERROR: PETSc Option Table entries: >>> [0]PETSC ERROR: -dm_plex_extrude_layers 3 >>> [0]PETSC ERROR: -dm_view vtk:mesh.vtk >>> [0]PETSC ERROR: -init_dm_plex_dim 2 >>> [0]PETSC ERROR: -petscpartitioner_type simple >>> [0]PETSC ERROR: -srf_dm_distribute >>> [0]PETSC ERROR: -srf_dm_refine 0 >>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>> error message to petsc-maint at mcs.anl.gov---------- >>> >>> Thanks, >>> Bhargav >>> >>> >>> ------------------------------ >>> This message and its contents, including attachments are intended solely >>> for the original recipient. If you are not the intended recipient or have >>> received this message in error, please notify me immediately and delete >>> this message from your computer system. Any unauthorized use or >>> distribution is prohibited. Please consider the environment before printing >>> this email. >> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> >> > > ------------------------------ > This message and its contents, including attachments are intended solely > for the original recipient. If you are not the intended recipient or have > received this message in error, please notify me immediately and delete > this message from your computer system. Any unauthorized use or > distribution is prohibited. Please consider the environment before printing > this email. -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Wed Nov 24 12:02:52 2021 From: knepley at gmail.com (Matthew Knepley) Date: Wed, 24 Nov 2021 13:02:52 -0500 Subject: [petsc-users] GAMG memory consumption In-Reply-To: <5B483CB3-9B0A-4B8C-B940-8CA8DD8AC135@stfc.ac.uk> References: <5B483CB3-9B0A-4B8C-B940-8CA8DD8AC135@stfc.ac.uk> Message-ID: On Wed, Nov 24, 2021 at 12:26 PM Karthikeyan Chockalingam - STFC UKRI < karthikeyan.chockalingam at stfc.ac.uk> wrote: > Hello, > > > > I would like to understand why more memory is consumed by -pc_type gamg > compared to -pc_type mg for the same problem size > > > > ksp/ksp/tutorial: ./ex45 -da_grid_x 368 -da_grid_x 368 -da_grid_x 368 > -ksp_type cg > > > > -pc_type mg > > > > Maximum (over computational time) process memory: total 1.9399e+10 > max 9.7000e+09 min 9.6992e+09 > > > > -pc_type gamg > > > > Maximum (over computational time) process memory: total 4.9671e+10 > max 2.4836e+10 min 2.4835e+10 > > > > > Am I right in understanding that the memory limiting factor is ?max > 2.4836e+10? as it is the maximum memory used at any given time? > Yes, I believe so. GAMG is using A_C = P^T A P, where P is the prolongation from coarse to fine, in order to compute the coarse operator A_C, rather than rediscretization, since it does not have any notion of discretization or coarse meshes. This takes more memory. Thanks, Matt > I have attached the -log_view output of both the preconditioners. > > > > Best regards, > > Karthik. > > > > This email and any attachments are intended solely for the use of the > named recipients. If you are not the intended recipient you must not use, > disclose, copy or distribute this email or any of its attachments and > should notify the sender immediately and delete this email from your > system. UK Research and Innovation (UKRI) has taken every reasonable > precaution to minimise risk of this email or any attachments containing > viruses or malware but the recipient should carry out its own virus and > malware checks before opening the attachments. UKRI does not accept any > liability for any losses or damages which the recipient may sustain due to > presence of any viruses. > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Wed Nov 24 12:18:44 2021 From: mfadams at lbl.gov (Mark Adams) Date: Wed, 24 Nov 2021 13:18:44 -0500 Subject: [petsc-users] GAMG memory consumption In-Reply-To: References: <5B483CB3-9B0A-4B8C-B940-8CA8DD8AC135@stfc.ac.uk> Message-ID: As Matt said GAMG uses more memory. But these numbers look odd: max == min and total = max + min, for both cases. I would use https://petsc.org/release/docs/manualpages/Sys/PetscMallocDump.html to look at this more closely. On Wed, Nov 24, 2021 at 1:03 PM Matthew Knepley wrote: > On Wed, Nov 24, 2021 at 12:26 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > >> Hello, >> >> >> >> I would like to understand why more memory is consumed by -pc_type gamg >> compared to -pc_type mg for the same problem size >> >> >> >> ksp/ksp/tutorial: ./ex45 -da_grid_x 368 -da_grid_x 368 -da_grid_x 368 >> -ksp_type cg >> >> >> >> -pc_type mg >> >> >> >> Maximum (over computational time) process memory: total 1.9399e+10 >> max 9.7000e+09 min 9.6992e+09 >> >> >> >> -pc_type gamg >> >> >> >> Maximum (over computational time) process memory: total 4.9671e+10 >> max 2.4836e+10 min 2.4835e+10 >> >> >> >> >> Am I right in understanding that the memory limiting factor is ?max >> 2.4836e+10? as it is the maximum memory used at any given time? >> > > Yes, I believe so. > > GAMG is using A_C = P^T A P, where P is the prolongation from coarse to > fine, in order to compute the coarse operator A_C, rather than > rediscretization, since it does not have any notion of discretization or > coarse meshes. This takes more memory. > > Thanks, > > Matt > > >> I have attached the -log_view output of both the preconditioners. >> >> >> >> Best regards, >> >> Karthik. >> >> >> >> This email and any attachments are intended solely for the use of the >> named recipients. If you are not the intended recipient you must not use, >> disclose, copy or distribute this email or any of its attachments and >> should notify the sender immediately and delete this email from your >> system. UK Research and Innovation (UKRI) has taken every reasonable >> precaution to minimise risk of this email or any attachments containing >> viruses or malware but the recipient should carry out its own virus and >> malware checks before opening the attachments. UKRI does not accept any >> liability for any losses or damages which the recipient may sustain due to >> presence of any viruses. >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave.mayhem23 at gmail.com Wed Nov 24 12:51:53 2021 From: dave.mayhem23 at gmail.com (Dave May) Date: Wed, 24 Nov 2021 19:51:53 +0100 Subject: [petsc-users] GAMG memory consumption In-Reply-To: References: <5B483CB3-9B0A-4B8C-B940-8CA8DD8AC135@stfc.ac.uk> Message-ID: I think your run with -pc_type mg is defining a multigrid hierarchy with a only single level. (A single level mg PC would also explain the 100+ iterations required to converge.) The gamg configuration is definitely coarsening your problem and has a deeper hierarchy. A single level hierarchy will require less memory than a multilevel hierarchy. Cheers, Dave On Wed 24. Nov 2021 at 19:03, Matthew Knepley wrote: > On Wed, Nov 24, 2021 at 12:26 PM Karthikeyan Chockalingam - STFC UKRI < > karthikeyan.chockalingam at stfc.ac.uk> wrote: > >> Hello, >> >> >> >> I would like to understand why more memory is consumed by -pc_type gamg >> compared to -pc_type mg for the same problem size >> >> >> >> ksp/ksp/tutorial: ./ex45 -da_grid_x 368 -da_grid_x 368 -da_grid_x 368 >> -ksp_type cg >> >> >> >> -pc_type mg >> >> >> >> Maximum (over computational time) process memory: total 1.9399e+10 >> max 9.7000e+09 min 9.6992e+09 >> >> >> >> -pc_type gamg >> >> >> >> Maximum (over computational time) process memory: total 4.9671e+10 >> max 2.4836e+10 min 2.4835e+10 >> >> >> >> >> Am I right in understanding that the memory limiting factor is ?max >> 2.4836e+10? as it is the maximum memory used at any given time? >> > > Yes, I believe so. > > GAMG is using A_C = P^T A P, where P is the prolongation from coarse to > fine, in order to compute the coarse operator A_C, rather than > rediscretization, since it does not have any notion of discretization or > coarse meshes. This takes more memory. > > Thanks, > > Matt > > >> I have attached the -log_view output of both the preconditioners. >> >> >> >> Best regards, >> >> Karthik. >> >> >> >> This email and any attachments are intended solely for the use of the >> named recipients. If you are not the intended recipient you must not use, >> disclose, copy or distribute this email or any of its attachments and >> should notify the sender immediately and delete this email from your >> system. UK Research and Innovation (UKRI) has taken every reasonable >> precaution to minimise risk of this email or any attachments containing >> viruses or malware but the recipient should carry out its own virus and >> malware checks before opening the attachments. UKRI does not accept any >> liability for any losses or damages which the recipient may sustain due to >> presence of any viruses. >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhargav.subramanya at kaust.edu.sa Wed Nov 24 16:45:19 2021 From: bhargav.subramanya at kaust.edu.sa (Bhargav Subramanya) Date: Thu, 25 Nov 2021 01:45:19 +0300 Subject: [petsc-users] dmplex extruded layers In-Reply-To: References: Message-ID: On Wed, Nov 24, 2021 at 8:59 PM Matthew Knepley wrote: > On Wed, Nov 24, 2021 at 12:27 PM Bhargav Subramanya < > bhargav.subramanya at kaust.edu.sa> wrote: > >> Dear Matt and Mark, >> >> Thank you very much for your reply. Your inputs are very useful to me. >> >> On Mon, Nov 22, 2021 at 9:38 PM Matthew Knepley >> wrote: >> >>> On Mon, Nov 22, 2021 at 12:10 PM Bhargav Subramanya < >>> bhargav.subramanya at kaust.edu.sa> wrote: >>> >>>> Dear All, >>>> >>>> I have a prismatic mesh generated by extruding a base icosahedron >>>> spherical surface mesh. The mesh is distributed on 2 processes after >>>> extrusion. The dof for my problem are defined only on the simplices of the >>>> spherical layers. >>>> >>>> 1. For a given spherical layer, I want to get points, which are >>>> basically simplices, lying on that layer only. DMPlexGetHeightStratum >>>> returns the points on all the spherical Layers. I can probably use >>>> DMPlexFindVertices (based on the radius of the spherical layer) and >>>> DMPlexGetSimplexOrBoxCells. Could you suggest if there is a better way to >>>> retrieve the points (which are simplices) on a given spherical layer in the >>>> extruded mesh? >>>> >>> >>> DMPlexGetHeightStratum() refers to height in the Hasse Diagram, which is >>> a DAG, not in the mesh. For example, height 0 points are the cells, height >>> 1 are the faces, etc. >>> >>> I believe the default numbering for extrusion (in the main branch), is >>> that all vertices produced from a given vertex be in order. This would mean >>> that if v were the vertex point number, then >>> >>> (v - vStart) % Nl == l >>> >>> where Nl is the number of layers and l is the layer of that vertex. It >>> is also the same for triangles. So if you want to segregate each shell, >>> after extrusion, make a label that gives triangles this marker, and then >>> use DMPlexLabelComplete(). Then after refinement you will still have the >>> shells labeled correctly. >>> >>> I would be happy to help you make an example that does this. It seems >>> cool. >>> >> >> ----- Sure, I am interested to make an example of it. And, I really >> appreciate your help. >> >>> >>> >>>> 2. I am able to visualize the entire mesh using dm_view. Is there a way >>>> to get the mesh file for the local dm from a specific process? >>>> >>> >>> You can use -dm_partition_view which outputs a field with the process >>> number. Then use Clip inside Paraview and clip to the process number you >>> want, >>> or just view that field so each process has a different color. >>> >>> >>>> 3. One of my previous emails might have got lost as I mistakenly >>>> attached a large mesh file and sent it. So I am repeating the question >>>> here. DMPlexExtrude gives the following error after distributing the base >>>> 2D spherical mesh. Both refinement or/and extrusion after dm distribution >>>> does not work for me. In fact, I tried >>>> with src/dm/impls/plex/tutorials/ex10.c also. Although the mesh is >>>> distributed after the extrusion in ex10.c (which is working fine for me), I >>>> tried to distribute before extrusion, and I get the following error. Could >>>> you please suggest where I might be making a mistake? >>>> >>> >>> So you want to distribute the mesh before extruding. For that small >>> example (using the main branch), I run >>> >>> mpiexec -n 3 ./meshtest -dm_plex_shape sphere -dm_refine_pre 2 >>> -dm_distribute -dm_partition_view -dm_view hdf5:mesh.h5 -dm_extrude 3 >>> >> >> ----- I pulled the code from the main branch and configured my petsc >> again. I am now finally able to reproduce the mesh that you have attached. >> I am able to do parallel mesh refinement, followed by extrusion, and use >> the following code to check that. However, I still have one problem. >> Attached are the extruded mesh files. >> >> 1. Mesh doesn't seem to be refined properly near interfaces between the >> processes. I then used -init_dm_distribute_overlap of 1 and that fixed the >> problem. I hope this is the way to do that. >> > > Hmm, something is wrong here. We should track this down and fix it. I do > not see this. Could you tell me how to reproduce the problem? > ----- I mainly wanted to check if the mesh was distributed or not before the extrusion. I do this by specifying different prefix options as shown in the code below. I think using different prefix options is probably causing the problem. #include int main(int argc, char **argv) { DM dm; PetscBool distributed; PetscErrorCode ierr; ierr = PetscInitialize(&argc, &argv, NULL, NULL);if (ierr) return ierr; ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, "init_");CHKERRQ(ierr); ierr = DMSetFromOptions(dm);CHKERRQ(ierr); ierr = DMPlexIsDistributed(dm, &distributed); ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); CHKERRQ(ierr); ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT); ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr); ierr = DMSetFromOptions(dm);CHKERRQ(ierr); ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); ierr = DMDestroy(&dm);CHKERRQ(ierr); ierr = PetscFinalize(); return ierr; } The command line options are: mpiexec -n 3 ./cavity_flow.out -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_extrude 3 -dm_partition_view -dm_view hdf5:mesh.h5 2. The parallel mesh refinement doesn't seem to take care of the spherical >> geometry (shown in the attached mesh file). Could you please suggest how to >> fix it? >> > > Okay, this is more subtle. Let me explain how things work inside. > > 1) When you refine, it inserts new cells/faces/edges/vertices. When we > insert a new vertex, we ask what the coordinates should be. If no special > information is there, we > just average the surrounding vertices. > > 2) Then we allow a second step that processed all coordinates in the mesh > after refinement > > For the sphere, we use 2) to make all points stick to the implicit surface > (radius R). We cannot do this for the extruded mesh, so we turn it off. > ---- thanks for this explanation. > What makes the most sense to me is to do > > 1) Serial mesh refinement to get enough cells to distribute evenly > > 2) Mesh distribution > > 3) Mesh refinement > > 4) Extrusion > > This should preserve the geometry and scale well. Does that sound right? > -- Yes, this sounds right, and this must preserve the geometry. In fact, I was under the assumption that I was doing the exact same thing by using these options: mpiexec -n 3 ./cavity_flow.out -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_extrude 3 -dm_partition_view -dm_view hdf5:mesh.h5 Do you think these options are not doing the things in the order of 1,2,3 and 4 as you specified? If not, could you suggest the options that I need to use to get in the order of 1,2,3 and 4? > > Thanks, > > Matt > > >> command line options: >> -init_dm_plex_shape sphere -init_dm_plex_sphere_radius 1.0 >> -init_dm_refine_pre 2 -init_dm_distribute -dm_refine 2 -dm_partition_view >> -dm_view hdf5:mesh.h5 -dm_extrude 3 >> >> ierr = DMCreate(PETSC_COMM_WORLD, &dm);CHKERRQ(ierr); >> ierr = DMSetType(dm, DMPLEX);CHKERRQ(ierr); >> ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, >> "init_");CHKERRQ(ierr); >> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >> >> ierr = DMPlexIsDistributed(dm, &distributed); >> ierr = PetscSynchronizedPrintf(PETSC_COMM_WORLD, "%d \n", distributed); >> CHKERRQ(ierr); >> ierr = PetscSynchronizedFlush(PETSC_COMM_WORLD,PETSC_STDOUT); >> >> ierr = PetscObjectSetOptionsPrefix((PetscObject) dm, NULL);CHKERRQ(ierr); >> ierr = DMSetFromOptions(dm);CHKERRQ(ierr); >> ierr = DMViewFromOptions(dm, NULL, "-dm_view");CHKERRQ(ierr); >> >> >>> and I get the attached picture. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: This DMPlex is distributed but its PointSF has no graph >>>> set >>>> [0]PETSC ERROR: See https://petsc.org/release/faq/ >>>> >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Release Version 3.16.1, unknown >>>> [0]PETSC ERROR: ./cavity_flow.out on a arch-darwin-c-debug named >>>> kl-21859 by subrambm Mon Nov 22 19:47:14 2021 >>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ >>>> --with-fc=gfortran --download-mpich --download-fblaslapack >>>> --download-superlu_dist --download-hypre --download-fiat >>>> --download-generator --download-triangle --download-tetgen --download-chaco >>>> --download-make -download-boost --download-cmake --download-ml >>>> --download-mumps=https://bitbucket.org/petsc/pkg-mumps.git >>>> >>>> --download-mumps-commit=v5.4.1-p1 --download-scalapack --download-ptscotch >>>> --download-hdf5 --force >>>> [0]PETSC ERROR: #1 DMPlexCheckPointSF() at >>>> /Users/subrambm/petsc/src/dm/impls/plex/plex.c:8554 >>>> [0]PETSC ERROR: #2 DMPlexOrientInterface_Internal() at >>>> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:595 >>>> [0]PETSC ERROR: #3 DMPlexInterpolate() at >>>> /Users/subrambm/petsc/src/dm/impls/plex/plexinterpolate.c:1357 >>>> [0]PETSC ERROR: #4 DMPlexExtrude() at >>>> /Users/subrambm/petsc/src/dm/impls/plex/plexcreate.c:1543 >>>> [0]PETSC ERROR: #5 CreateMesh() at ex10.c:161 >>>> [0]PETSC ERROR: #6 main() at ex10.c:180 >>>> [0]PETSC ERROR: PETSc Option Table entries: >>>> [0]PETSC ERROR: -dm_plex_extrude_layers 3 >>>> [0]PETSC ERROR: -dm_view vtk:mesh.vtk >>>> [0]PETSC ERROR: -init_dm_plex_dim 2 >>>> [0]PETSC ERROR: -petscpartitioner_type simple >>>> [0]PETSC ERROR: -srf_dm_distribute >>>> [0]PETSC ERROR: -srf_dm_refine 0 >>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire >>>> error message to petsc-maint at mcs.anl.gov---------- >>>> >>>> Thanks, >>>> Bhargav >>>> >>>> >>>> ------------------------------ >>>> This message and its contents, including attachments are intended >>>> solely for the original recipient. If you are not the intended recipient or >>>> have received this message in error, please notify me immediately and >>>> delete this message from your computer system. Any unauthorized use or >>>> distribution is prohibited. Please consider the environment before printing >>>> this email. >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >> >> ------------------------------ >> This message and its contents, including attachments are intended solely >> for the original recipient. If you are not the intended recipient or have >> received this message in error, please notify me immediately and delete >> this message from your computer system. Any unauthorized use or >> distribution is prohibited. Please consider the environment before printing >> this email. > > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rossi.f at inwind.it Thu Nov 25 04:15:03 2021 From: rossi.f at inwind.it (Fabio Rossi) Date: Thu, 25 Nov 2021 11:15:03 +0100 (CET) Subject: [petsc-users] configure error with fftw and without mpi Message-ID: <400981621.484717.1637835303203@mail1.libero.it> I am trying to compile petsc with the following options (Gentoo build system): ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64/petsc/lib64 scrollOutput=1 FFLAGS=-O2 -pipe -fPIC CFLAGS=-march=core2 -O2 -pipe -fPIC CXXFLAGS=-march=core2 -O2 -pipe -fPIC LDFLAGS=-Wl,-O1 -Wl,--as-needed --prefix=/usr/lib64/petsc --with-shared-libraries --with-single-library --with-clanguage=c --with-petsc-arch=linux-gnu-c-opt --with-precision=double --with-gnu-compilers --with-blas-lapack-lib=-lblas -llapack --with-debugging=0 --with-mpi=0 --with-cc=x86_64-pc-linux-gnu-gcc --with-cxx=x86_64-pc-linux-gnu-g++ --with-fortran=1 --with-fc=x86_64-pc-linux-gnu-gfortran --with-mpi-compilers=0 --with-scalar-type=real --with-windows-graphics=0 --with-matlab=0 --with-cmake:BOOL=1 --with-pthread=1 --with-afterimage=0 --without-hdf5 --with-hypre=0 --without-suitesparse --with-superlu=0 --with-x --with-x11 --with-ptscotch=0 --with- scalapack=0 --without-mumps --with-imagemagick=0 --with-python=0 --with-boost --with-fftw so MPI support is disabled, and I get the following configure error: TESTING: consistencyChecks from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:963) TESTING: checkDependencies from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:872) ******************************************************************************* UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): ------------------------------------------------------------------------------- Did not find package MPI needed by fftw. Enable the package using --with-mpi Isn't MPI optional? Where is the error coming from? If I disable fftw then it compiles successfully. Fabio From bsmith at petsc.dev Thu Nov 25 10:27:54 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 25 Nov 2021 11:27:54 -0500 Subject: [petsc-users] configure error with fftw and without mpi In-Reply-To: <400981621.484717.1637835303203@mail1.libero.it> References: <400981621.484717.1637835303203@mail1.libero.it> Message-ID: <405C70D4-B526-4109-80DB-96A645C7D9EB@petsc.dev> PETSc's install and use of fftw always requires MPI. The code in config/BuildSystem/config/packages/fftw.py and src/mat/impls/fft/fftw/fftw.c could relatively easily be "fixed" to also support building and using FFTW without MPI. Just change the fftw.py to conditionally pass in args.append('--enable-mpi') if self.mpi.found and change self.deps = [self.mpi,self.blasLapack] to self.deps = [self.blasLapack] self.odeps = [self.mpi] and in fftw.c add #if defined(PETSC_HAVE_MPI) around the parts of the code that depend on MPI. We'd welcome a merge request with these enhancements. Barry > On Nov 25, 2021, at 5:15 AM, Fabio Rossi via petsc-users wrote: > > I am trying to compile petsc with the following options (Gentoo build system): > > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64/petsc/lib64 scrollOutput=1 FFLAGS=-O2 -pipe -fPIC CFLAGS=-march=core2 -O2 -pipe -fPIC CXXFLAGS=-march=core2 -O2 -pipe -fPIC LDFLAGS=-Wl,-O1 -Wl,--as-needed --prefix=/usr/lib64/petsc --with-shared-libraries --with-single-library --with-clanguage=c --with-petsc-arch=linux-gnu-c-opt --with-precision=double --with-gnu-compilers --with-blas-lapack-lib=-lblas -llapack --with-debugging=0 --with-mpi=0 --with-cc=x86_64-pc-linux-gnu-gcc --with-cxx=x86_64-pc-linux-gnu-g++ --with-fortran=1 --with-fc=x86_64-pc-linux-gnu-gfortran --with-mpi-compilers=0 --with-scalar-type=real --with-windows-graphics=0 --with-matlab=0 --with-cmake:BOOL=1 --with-pthread=1 --with-afterimage=0 --without-hdf5 --with-hypre=0 --without-suitesparse --with-superlu=0 --with-x --with-x11 --with-ptscotch=0 > --with- > scalapack=0 --without-mumps --with-imagemagick=0 --with-python=0 --with-boost --with-fftw > > so MPI support is disabled, and I get the following configure error: > > TESTING: consistencyChecks from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:963) > TESTING: checkDependencies from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:872) > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Did not find package MPI needed by fftw. > Enable the package using --with-mpi > > Isn't MPI optional? Where is the error coming from? > > If I disable fftw then it compiles successfully. > > Fabio From rossi.f at inwind.it Thu Nov 25 13:29:31 2021 From: rossi.f at inwind.it (Fabio Rossi) Date: Thu, 25 Nov 2021 20:29:31 +0100 (CET) Subject: [petsc-users] configure error with fftw and without mpi In-Reply-To: <405C70D4-B526-4109-80DB-96A645C7D9EB@petsc.dev> References: <400981621.484717.1637835303203@mail1.libero.it> <405C70D4-B526-4109-80DB-96A645C7D9EB@petsc.dev> Message-ID: <526954230.547904.1637868571826@mail1.libero.it> Thanks for the reply. I am using the system fftw which is provided by the distribution so I am not going to install it by using PETSC's install scripts. I am sorry but I am not confident with the PETSC building system. What happens if mpi is installed but the user prefers to compile PETSC with --with-mpi=0 configure option? In that case self.mpi.found would be true? Regarding fftw.c, basically the functions to modify are: extern PetscErrorCode MatMult_SeqFFTW(Mat,Vec,Vec); extern PetscErrorCode MatMultTranspose_SeqFFTW(Mat,Vec,Vec); extern PetscErrorCode MatMult_MPIFFTW(Mat,Vec,Vec); extern PetscErrorCode MatMultTranspose_MPIFFTW(Mat,Vec,Vec); extern PetscErrorCode MatDestroy_FFTW(Mat); extern PetscErrorCode VecDestroy_MPIFFTW(Vec); extern PetscErrorCode MatCreateVecsFFTW_FFTW(Mat,Vec*,Vec*,Vec*); PETSC_EXTERN PetscErrorCode MatCreate_FFTW(Mat A) The *MPI* function should exist only with PETSC_HAVE_MPI true while inside the other functions the number of processors (detected inside the variable "size") should be initialized to 1 and support the parallel case only when PETSC_HAVE_MPI is true. As a user of the PETSC library and not programmer (I need to build another library which requires it), I want to put your attention to the documentation at https://petsc.org/release/install/install/#doc-config-externalpack where it's written "PETSc may be built and run without MPI support if processing only in serial." which then seems to be true for PETSc itself but not for its dependencies because the interface to fftw, right now, depends on MPI. That's the reason of this new thread in the mailing list :-) Fabio > Il 25/11/2021 17:27 Barry Smith ha scritto: > > > PETSc's install and use of fftw always requires MPI. > > The code in config/BuildSystem/config/packages/fftw.py and src/mat/impls/fft/fftw/fftw.c could relatively easily be "fixed" to also support building and using FFTW without MPI. > > Just change the fftw.py to conditionally pass in args.append('--enable-mpi') if self.mpi.found and change > > self.deps = [self.mpi,self.blasLapack] > > to > > self.deps = [self.blasLapack] > self.odeps = [self.mpi] > > and in fftw.c add #if defined(PETSC_HAVE_MPI) around the parts of the code that depend on MPI. > > We'd welcome a merge request with these enhancements. > > Barry > > > > On Nov 25, 2021, at 5:15 AM, Fabio Rossi via petsc-users wrote: > > > > I am trying to compile petsc with the following options (Gentoo build system): > > > > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64/petsc/lib64 scrollOutput=1 FFLAGS=-O2 -pipe -fPIC CFLAGS=-march=core2 -O2 -pipe -fPIC CXXFLAGS=-march=core2 -O2 -pipe -fPIC LDFLAGS=-Wl,-O1 -Wl,--as-needed --prefix=/usr/lib64/petsc --with-shared-libraries --with-single-library --with-clanguage=c --with-petsc-arch=linux-gnu-c-opt --with-precision=double --with-gnu-compilers --with-blas-lapack-lib=-lblas -llapack --with-debugging=0 --with-mpi=0 --with-cc=x86_64-pc-linux-gnu-gcc --with-cxx=x86_64-pc-linux-gnu-g++ --with-fortran=1 --with-fc=x86_64-pc-linux-gnu-gfortran --with-mpi-compilers=0 --with-scalar-type=real --with-windows-graphics=0 --with-matlab=0 --with-cmake:BOOL=1 --with-pthread=1 --with-afterimage=0 --without-hdf5 --with-hypre=0 --without-suitesparse --with-superlu=0 --with-x --with-x11 --with-ptscot ch=0 > > --with- > > scalapack=0 --without-mumps --with-imagemagick=0 --with-python=0 --with-boost --with-fftw > > > > so MPI support is disabled, and I get the following configure error: > > > > TESTING: consistencyChecks from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:963) > > TESTING: checkDependencies from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:872) > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > Did not find package MPI needed by fftw. > > Enable the package using --with-mpi > > > > Isn't MPI optional? Where is the error coming from? > > > > If I disable fftw then it compiles successfully. > > > > Fabio From knepley at gmail.com Thu Nov 25 13:52:06 2021 From: knepley at gmail.com (Matthew Knepley) Date: Thu, 25 Nov 2021 14:52:06 -0500 Subject: [petsc-users] configure error with fftw and without mpi In-Reply-To: <526954230.547904.1637868571826@mail1.libero.it> References: <400981621.484717.1637835303203@mail1.libero.it> <405C70D4-B526-4109-80DB-96A645C7D9EB@petsc.dev> <526954230.547904.1637868571826@mail1.libero.it> Message-ID: On Thu, Nov 25, 2021 at 2:29 PM Fabio Rossi via petsc-users < petsc-users at mcs.anl.gov> wrote: > Thanks for the reply. > > I am using the system fftw which is provided by the distribution so I am > not going to install it by using PETSC's install scripts. > > I am sorry but I am not confident with the PETSC building system. What > happens if mpi is installed but the user prefers to compile PETSC with > --with-mpi=0 configure option? In that case self.mpi.found would be true? > > Regarding fftw.c, basically the functions to modify are: > > extern PetscErrorCode MatMult_SeqFFTW(Mat,Vec,Vec); > extern PetscErrorCode MatMultTranspose_SeqFFTW(Mat,Vec,Vec); > extern PetscErrorCode MatMult_MPIFFTW(Mat,Vec,Vec); > extern PetscErrorCode MatMultTranspose_MPIFFTW(Mat,Vec,Vec); > extern PetscErrorCode MatDestroy_FFTW(Mat); > extern PetscErrorCode VecDestroy_MPIFFTW(Vec); > extern PetscErrorCode MatCreateVecsFFTW_FFTW(Mat,Vec*,Vec*,Vec*); > PETSC_EXTERN PetscErrorCode MatCreate_FFTW(Mat A) > > The *MPI* function should exist only with PETSC_HAVE_MPI true while inside > the other functions the number of processors (detected inside the variable > "size") should be initialized to 1 and support the parallel case only when > PETSC_HAVE_MPI is true. > > As a user of the PETSC library and not programmer (I need to build another > library which requires it), I want to put your attention to the > documentation at > > https://petsc.org/release/install/install/#doc-config-externalpack > > where it's written "PETSc may be built and run without MPI support if > processing only in serial." > This is true, but we cannot control our dependencies. Thanks, Matt > which then seems to be true for PETSc itself but not for its dependencies > because the interface to fftw, right now, depends on MPI. That's the reason > of this new thread in the mailing list :-) > > Fabio > > > Il 25/11/2021 17:27 Barry Smith ha scritto: > > > > > > PETSc's install and use of fftw always requires MPI. > > > > The code in config/BuildSystem/config/packages/fftw.py and > src/mat/impls/fft/fftw/fftw.c could relatively easily be "fixed" to also > support building and using FFTW without MPI. > > > > Just change the fftw.py to conditionally pass in > args.append('--enable-mpi') if self.mpi.found and change > > > > self.deps = [self.mpi,self.blasLapack] > > > > to > > > > self.deps = [self.blasLapack] > > self.odeps = [self.mpi] > > > > and in fftw.c add #if defined(PETSC_HAVE_MPI) around the parts of the > code that depend on MPI. > > > > We'd welcome a merge request with these enhancements. > > > > Barry > > > > > > > On Nov 25, 2021, at 5:15 AM, Fabio Rossi via petsc-users < > petsc-users at mcs.anl.gov> wrote: > > > > > > I am trying to compile petsc with the following options (Gentoo build > system): > > > > > > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu > --host=x86_64-pc-linux-gnu --mandir=/usr/share/man > --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc > --localstatedir=/var/lib --libdir=/usr/lib64/petsc/lib64 scrollOutput=1 > FFLAGS=-O2 -pipe -fPIC CFLAGS=-march=core2 -O2 -pipe -fPIC > CXXFLAGS=-march=core2 -O2 -pipe -fPIC LDFLAGS=-Wl,-O1 -Wl,--as-needed > --prefix=/usr/lib64/petsc --with-shared-libraries --with-single-library > --with-clanguage=c --with-petsc-arch=linux-gnu-c-opt > --with-precision=double --with-gnu-compilers --with-blas-lapack-lib=-lblas > -llapack --with-debugging=0 --with-mpi=0 --with-cc=x86_64-pc-linux-gnu-gcc > --with-cxx=x86_64-pc-linux-gnu-g++ --with-fortran=1 > --with-fc=x86_64-pc-linux-gnu-gfortran --with-mpi-compilers=0 > --with-scalar-type=real --with-windows-graphics=0 --with-matlab=0 > --with-cmake:BOOL=1 --with-pthread=1 --with-afterimage=0 --without-hdf5 > --with-hypre=0 --without-suitesparse --with-superlu=0 --with-x --with-x11 > --with-ptscot > ch=0 > > > --with- > > > scalapack=0 --without-mumps --with-imagemagick=0 --with-python=0 > --with-boost --with-fftw > > > > > > so MPI support is disabled, and I get the following configure error: > > > > > > TESTING: consistencyChecks from > config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:963) > > > TESTING: checkDependencies from > config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:872) > > > > ******************************************************************************* > > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log > for details): > > > > ------------------------------------------------------------------------------- > > > Did not find package MPI needed by fftw. > > > Enable the package using --with-mpi > > > > > > Isn't MPI optional? Where is the error coming from? > > > > > > If I disable fftw then it compiles successfully. > > > > > > Fabio > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Thu Nov 25 22:15:33 2021 From: bsmith at petsc.dev (Barry Smith) Date: Thu, 25 Nov 2021 23:15:33 -0500 Subject: [petsc-users] configure error with fftw and without mpi In-Reply-To: <400981621.484717.1637835303203@mail1.libero.it> References: <400981621.484717.1637835303203@mail1.libero.it> Message-ID: I have made a branch barry/2021-11-25/feature-fftw-seq https://gitlab.com/petsc/petsc/-/merge_requests/4604 that provides support for using FFTW without MPI, please let us know if you have difficulties with it. Barry > On Nov 25, 2021, at 5:15 AM, Fabio Rossi via petsc-users wrote: > > I am trying to compile petsc with the following options (Gentoo build system): > > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64/petsc/lib64 scrollOutput=1 FFLAGS=-O2 -pipe -fPIC CFLAGS=-march=core2 -O2 -pipe -fPIC CXXFLAGS=-march=core2 -O2 -pipe -fPIC LDFLAGS=-Wl,-O1 -Wl,--as-needed --prefix=/usr/lib64/petsc --with-shared-libraries --with-single-library --with-clanguage=c --with-petsc-arch=linux-gnu-c-opt --with-precision=double --with-gnu-compilers --with-blas-lapack-lib=-lblas -llapack --with-debugging=0 --with-mpi=0 --with-cc=x86_64-pc-linux-gnu-gcc --with-cxx=x86_64-pc-linux-gnu-g++ --with-fortran=1 --with-fc=x86_64-pc-linux-gnu-gfortran --with-mpi-compilers=0 --with-scalar-type=real --with-windows-graphics=0 --with-matlab=0 --with-cmake:BOOL=1 --with-pthread=1 --with-afterimage=0 --without-hdf5 --with-hypre=0 --without-suitesparse --with-superlu=0 --with-x --with-x11 --with-ptscotch=0 > --with- > scalapack=0 --without-mumps --with-imagemagick=0 --with-python=0 --with-boost --with-fftw > > so MPI support is disabled, and I get the following configure error: > > TESTING: consistencyChecks from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:963) > TESTING: checkDependencies from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:872) > ******************************************************************************* > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > ------------------------------------------------------------------------------- > Did not find package MPI needed by fftw. > Enable the package using --with-mpi > > Isn't MPI optional? Where is the error coming from? > > If I disable fftw then it compiles successfully. > > Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Thu Nov 25 23:48:42 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Fri, 26 Nov 2021 14:48:42 +0900 Subject: [petsc-users] DMPlex shared node indexing Message-ID: Dear PETSc-team, When the mesh is distributed, the vertices are renumbered and some vertices are shared by neighboring CPUs. My question is: 1. How to check which vertice is a ghost one? 2. Are those ghost vertices always at the end of vertices list? If not 3. Are there simple ways to let those ghost vertices come after owned ones? Many thanks Yuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhfreewill at gmail.com Fri Nov 26 01:13:46 2021 From: zhfreewill at gmail.com (zhfreewill) Date: Fri, 26 Nov 2021 15:13:46 +0800 Subject: [petsc-users] How to apply boundary conditions when using TS? In-Reply-To: References: Message-ID: Thank you for your reply. This is a good FD example for me and I think I understood it at least for FD discretization. Best, Qw Matthew Knepley ?2021?11?16??? ??7:11??? > On Mon, Nov 15, 2021 at 10:24 PM zhfreewill wrote: > >> Hi, >> I'm new to PETSc. I have been confused about how to apply boundary >> conditions (BCs) (e.g., the Dirichlet and the Neumann type) when using the >> time-stepping (TS) object of PETSc. >> >> I have read the example codes ( >> https://petsc.org/release/src/ts/tutorials/index.html) and vaguely found >> that BCs seems to be set as follows: >> >> 1. when evaluating a matrix of the right hand side function g(x) (e.g., >> using a function like RHSMatrixHeat) >> 2. when evaluating a matrix of the Jacobian matrix of g'(x) (e.g., using >> a function like FormJacobian) >> >> For instance, in the ex1.4 ( >> https://petsc.org/release/src/ts/tutorials/ex4.c.html), with comments at >> the beginning states >> >> 17: /* >> ------------------------------------------------------------------------ >> 19: This program solves the one-dimensional heat equation (also >> called the >> 20: diffusion equation), >> 21: u_t = u_xx, >> 22: on the domain 0 <= x <= 1, with the boundary conditions >> 23: u(t,0) = 0, u(t,1) = 0, >> 24: and the initial condition >> 25: u(0,x) = sin(6*pi*x) + 3*sin(2*pi*x). >> 26: This is a linear, second-order, parabolic equation. >> >> 28: We discretize the right-hand side using finite differences with >> 29: uniform grid spacing h: >> 30: u_xx = (u_{i+1} - 2u_{i} + u_{i-1})/(h^2) >> ... >> 47: >> ------------------------------------------------------------------------- */ >> >> and Dirichlet BCs, u(t,0) = 0 and u(t,1) = 0, are to set on the both >> side of the discretized domain, the corresponding codes can be found on the >> following lines in a function RHSMatrixHeat: >> >> /* >> ------------------------------------------------------------------------- */ >> 463: RHSMatrixHeat - User-provided routine to compute the right-hand-side >> matrix for the heat equation. */ >> 491: PetscErrorCode RHSMatrixHeat(TS ts,PetscReal t,Vec X,Mat AA,Mat >> BB,void *ctx) >> 492: { >> ... >> 505: /* Set matrix rows corresponding to boundary data */ >> 509: if (mstart == 0) { /* first processor only */ >> 510: v[0] = 1.0; >> 511: MatSetValues(A,1,&mstart,1,&mstart,v,INSERT_VALUES); >> 512: mstart++; >> 513: } >> 515: if (mend == appctx->m) { /* last processor only */ >> 516: mend--; >> 517: v[0] = 1.0; >> 518: MatSetValues(A,1,&mend,1,&mend,v,INSERT_VALUES); >> 519: } >> >> 521: /* Set matrix rows corresponding to interior data. We construct >> the matrix one row at a time. */ >> 525: v[0] = sone; v[1] = stwo; v[2] = sone; >> 526: for (i=mstart; i> 527: idx[0] = i-1; idx[1] = i; idx[2] = i+1; >> 528: MatSetValues(A,1,&i,3,idx,v,INSERT_VALUES); >> 529: } >> ... >> 550: } >> /* >> ------------------------------------------------------------------------- */ >> >> >> My questions are: >> >> 1. How do these lines of code embody the Dirichlet BCs u(t,0) = 0 and >> u(t,1) = 0. >> > > This is the finite difference version of BC. It is just moving the term > for a known value to the forcing. However, since > the BC is 0, then we just do not compute the term. If the BC were nonzero, > you would also see the term in the > residual. For example, > > > https://gitlab.com/petsc/petsc/-/blob/main/src/snes/tutorials/ex19.c#L271 > > >> 2. Similarly, if I want to apply a Neumann type BCs in these lines, How >> can I do? >> > > For finite differences, you usually just approximate the derivative as the > one-sided difference of the last two values, > and set this equal to the known value. For finite elements, Neumann > conditions are just an addition weak form > integrated over the boundary. > > Thanks, > > Matt > > >> Any suggestions or documents? >> >> Best, >> Qw >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhfreewill at gmail.com Fri Nov 26 03:02:21 2021 From: zhfreewill at gmail.com (zhfreewill) Date: Fri, 26 Nov 2021 17:02:21 +0800 Subject: [petsc-users] How to set boundary conditions for a structured finite volume (FV) discretization when using TS/SNES? In-Reply-To: References: Message-ID: Hi all, I have a basic question on how to apply B.C. for a cell-centered structured FV discretization when using the SNES and Ts object. For instance, for a 1-D transient diffusion equation *u_t = u_xx, 0 < x < 1* with IC: *u(x,t=0) = 0;* and Dirichlet BC: *u(x=0) = u_A = 0.0;* *u(x=1) = u_B = 1.0;* or Neumann BC: *du/dx(x=0.0) = q_A = 0.0;* *du/dx(x=1.0) = q_B = 1.0;* When using TS of PETSc, the equation can be organized in form of *U_t = G(t,u)* where *G* is the right-hand-side (RHS) function regardless of the specific method of discretization. *(1) For FD discretization* *A |---dx--| B* *|-------|-------|-------|-------|-------| grid size = nx* *0 i-1 i i+1 nx-1* the vertex-centered FD discretization of the domain is *x(i) = i*dx (i = 0,..., nx)), where dx = 1/nx* and the RHS function G of the equation is *G[i] = (u[i+1]-2*u[i]+u[i-1])/dx^2* There have been dozens of examples demonstrating way of BC setting with a finite difference medthod, e.g., https://github.com/petsc/petsc/blob/386ebab930a846575609b49074a40c46e7f8ed75/src/ts/tutorials/ex26.c#L293-L343 which I think I have understood how it works and the following lines of code evaluate the RHS function *G(t,u)* in PETSc's *TSSetRHSFunction()* routine /*------------------FD discretization-------------------------*/ *for (i = xs; i < xs + xm; i++) {* *// left boundary* *if (i == 0) {* *G[i] = u[i] - u_A; **// Dirichlet BC: u(x=0) = u_A;* *G[i] = (u[i] - u[i+1])/dx - q_A; **// Neumann BC: du/dx(x=0.0) = q_A;* *} * *// right boundary* *else if (i == mx - 1) {* *G[i] = u[i] - u_B; **// Dirichlet BC: u(x=1.0) = u_B;* *G[i] = (u[i]-u[i-1])/dx - q_B; **// Neumann BC: du/dx(x=0.0) = q_A;* *}* *// internal* *else {* *G[i] = (u[i-1] - 2 * u[i] + u[i + 1])/dx^2; **//u_xx* *}* *}**/*------------------------------------------------------------*/* The BCs, e.g., at the left boudnary, can be straightforwardly set as follows: Dirichlet BC: *G[0] = u[0] - u_A* which indicates *G[0] = u[0] - u_A = 0* and Dirichlet BC *u[0] = u_A* is thus applied Neumann BC: *G[0] = (u[0] - u[1])/dx - q_A* which indicates *G[0] = (u[0] - u[1])/dx - q_A = 0* and Neumann BC *du/dx = (u[0] - u[1])/dx = q_A* is applied This is the way the above examples apply BCs. For a structured, cell-centered FVM discretization, however, I tried and found the BCs can not be applied in a similar way. I will explain how I tried this as follows. *(2) FV discretization* *A |---dx--| B* *|---o---|---o---|---o---|---o---|---o---| cell size = nx* *W w P e E * *0 i-1 i i+1 nx-1* (NOTE: the center cell is denoted as P and its neighbouring cells as W, E; cell faces are denoted as w, e; the left and right boundary faces are A and B, respectively.) The cell-centered FV discretization of the domain is *x(i) = dx/2 + i*dx (i = 0,..., nx-1)), where dx = 1/nx* the diffusion term u_xx can be reorganized by using divergence theorem *(u_E-u_P) (u_P-u_W)* *--------- - ---------* * dx dx* and the RHS function G of the equation is *G[i] = (u_E-u_P)/dx - (u_P-u_W)/dx = (u[i+1]-2*u[i]+u[i-1])/dx* the key lines of code in RHS function G(t,u) in PETSc's *TSSetRHSFunction()* routine can be */*------------------FV discretization-------------------------*/* *for (i = xs; i < xs + xm; i++) {* *// left boundary* *if (i == 0) {* *G[i] = 3.0 * u[i] - u[i+1] - 2.0 * u_A; **//Dirichlet BC: u(x=0.0) = u_A, does not work* *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) = q_A;* *}* *// right boundary* *else if (i == mx - 1) {* *G[i] = 3.0 * u[i] - u[i-1] - 2.0 * u_B; **//Dirichlet BC: u(x=1.0) = u_B, does not work* *G[i] = (u[i]-u[i-1])/dx - q_B; **//Neumann BC: du/dx(x=1.0) = q_B, does not work* *}* *// internal* *else {* *G[i] = (u[i-1] - 2*u[i] + u[i+1])/dx; **//u_x* *}* *}**/*------------------------------------------------------------*/* I tried to apply BCs in a way similar to that of the FD discretization. For the Dirichlet BC on the left boundary, because of the cell-centered arrangement of u[i], the left BC is given as *u_A* at cell face A instead of cell center. So, I first applied a linear interpolation to get the cell-centered value of *u[0]* from the left boundary value *u_A* and neighbouring *u[1]* *u[0] = (u[1]+2*u_A)/3.0 (equivalent to 3*u[0] - u[1] - 2*u_A = 0.0)* * |-dx/2-|------dx-----|* * |------o------|------o------| linear interpolation: u[0] = (u[1]+2*u_A)/3.0* * A P e E* * u_A u[0] u[1]* Dirichlet BC: *G[0] = 3.0*u[0] - u[1] - 2.0*u_A* in which *u[0] = (u[1]+2*u_A)/3.0* is implicitly calculated and I expect the Dirichlet BC *u(x=0) = u_A* is applied For the Neumann BC, left BC is given as a gradient of *u*, *q_A*, at cell face A Neumann BC: *G[i] = (u[1] - u[0])/dx - q_A* in which Neumann BC *du/dx = (u[1] - u[0])/dx = q_A* is expected to be applied. However, this FV discretization failed to work. Is there anything wrong or could you give me a hint? Best, Qw -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Fri Nov 26 07:16:55 2021 From: mfadams at lbl.gov (Mark Adams) Date: Fri, 26 Nov 2021 08:16:55 -0500 Subject: [petsc-users] DMPlex shared node indexing In-Reply-To: References: Message-ID: First, ghost vertices only show up in "local" (vs "global") vectors and the ghost values are ordered after the locally owned ones. Others might have more to add. Mark On Fri, Nov 26, 2021 at 12:49 AM ?? wrote: > Dear PETSc-team, > > When the mesh is distributed, the vertices are renumbered and some > vertices are shared by neighboring CPUs. My question is: > > 1. How to check which vertice is a ghost one? > 2. Are those ghost vertices always at the end of vertices list? If not > 3. Are there simple ways to let those ghost vertices come after owned ones? > > Many thanks > > Yuan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.scott at epcc.ed.ac.uk Fri Nov 26 09:10:23 2021 From: d.scott at epcc.ed.ac.uk (David Scott) Date: Fri, 26 Nov 2021 15:10:23 +0000 Subject: [petsc-users] Error Writing Large HDF5 File Message-ID: Hello, I am trying to write out HDF5 files. The program I have works all right up to a point but then fails when I double the size of the file that I am trying to write out. I have attached a file containing the error message and another file containing the short program that I used to generate it. The first of these files reports that Petsc 3.14.2 was used. I know that this is old but I have obtained the same result with Petsc 3.16.1. Also, I have observed the same behaviour on two different machines. I tried using 64 bit indices but it made no difference. The program runs successfully with the following command line options -global_dim_x 1344 -global_dim_y 336 -global_dim_z 336 but fails with these -global_dim_x 2688 -global_dim_y 336 -global_dim_z 336 All the best, David The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336. -------------- next part -------------- -global_dim_x = 2688 -global_dim_y = 336 -global_dim_z = 336 Writing 1 Writing 2 Writing 3 HDF5-DIAG: Error detected in HDF5 (1.12.0) MPI-process 0: #000: ../../src/H5D.c line 151 in H5Dcreate2(): unable to create dataset major: Dataset minor: Unable to initialize object #001: ../../src/H5VLcallback.c line 1869 in H5VL_dataset_create(): dataset create failed major: Virtual Object Layer minor: Unable to create file #002: ../../src/H5VLcallback.c line 1835 in H5VL__dataset_create(): dataset create failed major: Virtual Object Layer minor: Unable to create file #003: ../../src/H5VLnative_dataset.c line 75 in H5VL__native_dataset_create(): unable to create dataset major: Dataset minor: Unable to initialize object #004: ../../src/H5Dint.c line 411 in H5D__create_named(): unable to create and link to dataset major: Dataset minor: Unable to initialize object #005: ../../src/H5L.c line 1804 in H5L_link_object(): unable to create new link to object major: Links minor: Unable to initialize object #006: ../../src/H5L.c line 2045 in H5L__create_real(): can't insert link major: Links minor: Unable to insert object #007: ../../src/H5Gtraverse.c line 855 in H5G_traverse(): internal path traversal failed major: Symbol table minor: Object not found #008: ../../src/H5Gtraverse.c line 630 in H5G__traverse_real(): traversal operator failed major: Symbol table minor: Callback failed #009: ../../src/H5L.c line 1851 in H5L__link_cb(): unable to create object major: Links minor: Unable to initialize object #010: ../../src/H5Oint.c line 2522 in H5O_obj_create(): unable to open object major: Object header minor: Can't open object #011: ../../src/H5Doh.c line 301 in H5O__dset_create(): unable to create dataset major: Dataset minor: Unable to initialize object #012: ../../src/H5Dint.c line 1374 in H5D__create(): can't update the metadata cache major: Dataset minor: Unable to initialize object #013: ../../src/H5Dint.c line 1071 in H5D__update_oh_info(): unable to update layout/pline/efl header message major: Dataset minor: Unable to initialize object #014: ../../src/H5Dlayout.c line 508 in H5D__layout_oh_create(): unable to initialize storage major: Dataset minor: Unable to initialize object #015: ../../src/H5Dint.c line 2387 in H5D__alloc_storage(): unable to initialize dataset with fill value major: Dataset minor: Unable to initialize object #016: ../../src/H5Dint.c line 2474 in H5D__init_storage(): unable to allocate all chunks of dataset major: Dataset minor: Unable to initialize object #017: ../../src/H5Dchunk.c line 4737 in H5D__chunk_allocate(): unable to write raw data to file major: Low-level I/O minor: Write failed #018: ../../src/H5Dchunk.c line 5081 in H5D__chunk_collective_fill(): MPI_Type_free failed major: Internal error (too specific to document in detail) minor: Some MPI function failed #019: ../../src/H5Dchunk.c line 5081 in H5D__chunk_collective_fill(): Invalid datatype, error stack: PMPI_Type_free(149): MPI_Type_free(datatype_p=0x7fff23a2b938) failed PMPI_Type_free(83).: Invalid datatype major: Internal error (too specific to document in detail) minor: MPI Error String #020: ../../src/H5Dchunk.c line 5079 in H5D__chunk_collective_fill(): MPI_Type_free failed major: Internal error (too specific to document in detail) minor: Some MPI function failed #021: ../../src/H5Dchunk.c line 5079 in H5D__chunk_collective_fill(): Invalid datatype, error stack: PMPI_Type_free(149): MPI_Type_free(datatype_p=0x7fff23a2b940) failed PMPI_Type_free(83).: Invalid datatype major: Internal error (too specific to document in detail) minor: MPI Error String #022: ../../src/H5Dchunk.c line 5038 in H5D__chunk_collective_fill(): MPI_Type_create_hindexed failed major: Internal error (too specific to document in detail) minor: Some MPI function failed #023: ../../src/H5Dchunk.c line 5038 in H5D__chunk_collective_fill(): Invalid argument, error stack: PMPI_Type_create_hindexed(143): MPI_Type_create_hindexed(count=1, array_of_blocklengths=0x1eba120, array_of_displacements=0x1eba140, MPI_BYTE, newtype=0x7fff23a2b940) failed PMPI_Type_create_hindexed(96).: Invalid value for blocklength, must be non-negative but is -1860026368 major: Internal error (too specific to document in detail) minor: MPI Error String [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HDF5 call H5Dcreate2() Status -1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.14.2, Dec 03, 2020 [0]PETSC ERROR: hdf5.x on a named nid005829 by dmse643 Fri Nov 26 12:54:34 2021 [0]PETSC ERROR: Configure options --known-has-attribute-aligned=1 --known-mpi-int64_t=0 --known-bits-per-byte=8 --known-64-bit-blas-indices=0 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 --known-level1-dcache-assoc=4 --known-level1-dcache-linesize=64 --known-level1-dcache-size=16384 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0 --with-fc=ftn --with-fortran-datatypes=1 --with-fortran-interfaces=1 --with-fortran-bindings=1 --with-fortranlib-autodetect=0 --with-ranlib=ranlib --with-scalar-type=real --with-shared-ld=ar --with-etags=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0 --with-mpi-lib="[]" --with-mpi-include="[]" --with-mpiexec=srun --with-blaslapack=1 --with-superlu=1 --with-superlu_dist=1 --with-parmetis=1 --with-metis=1 --with-hypre=1 --with-scalapack=1 --with-ptscotch=1 --with-ptscotch-include= --with-ptscotch-lib= --with-mumps=1 --with-mumps-include= --with-mumps-lib=-lmpifort --with-hdf5=1 --CFLAGS="-O3 -ffast-math -fPIC " --CPPFLAGS="-I/work/y07/shared/libs/core/petsc/3.14.2/GNU/9.3/include " --CXXFLAGS="-O3 -ffast-math -fPIC " --with-cxx-dialect=C++11 --FFLAGS="-O3 -ffast-math -fPIC " --LDFLAGS="-L/work/y07/shared/libs/core/petsc/3.14.2/GNU/9.3/lib " --LIBS="-lgfortran -lgcc -lstdc++" --PETSC_ARCH=x86-rome --prefix=/work/y07/shared/libs/core/petsc/3.14.2/GNU/9.3 [0]PETSC ERROR: #1 VecView_MPI_HDF5_DA() line 518 in /mnt/lustre/a2fs-work1/work/y07/y07/cse/pe-scripts/petsc-3.14.2/src/dm/impls/da/gr2.c [0]PETSC ERROR: #2 VecView_MPI_DA() line 698 in /mnt/lustre/a2fs-work1/work/y07/y07/cse/pe-scripts/petsc-3.14.2/src/dm/impls/da/gr2.c [0]PETSC ERROR: #3 VecView() line 608 in /mnt/lustre/a2fs-work1/work/y07/y07/cse/pe-scripts/petsc-3.14.2/src/vec/vec/interface/vector.c Writing 4 HDF5-DIAG: Error detected in HDF5 (1.12.0) MPI-process 0: #000: ../../src/H5F.c line 886 in H5Fclose(): decrementing file ID failed major: Object atom minor: Unable to close file #001: ../../src/H5I.c line 1422 in H5I_dec_app_ref(): can't decrement ID ref count major: Object atom minor: Unable to decrement reference count #002: ../../src/H5F.c line 243 in H5F__close_cb(): unable to close file major: File accessibility minor: Unable to close file #003: ../../src/H5VLcallback.c line 3977 in H5VL_file_close(): file close failed major: Virtual Object Layer minor: Unable to close file #004: ../../src/H5VLcallback.c line 3945 in H5VL__file_close(): file close failed major: Virtual Object Layer minor: Unable to close file #005: ../../src/H5VLnative_file.c line 884 in H5VL__native_file_close(): can't close file major: File accessibility minor: Unable to decrement reference count #006: ../../src/H5Fint.c line 2049 in H5F__close(): can't close file, there are objects still open major: File accessibility minor: Unable to close file [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Error in external library [0]PETSC ERROR: Error in HDF5 call H5Fclose() Status -1 [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Release Version 3.14.2, Dec 03, 2020 [0]PETSC ERROR: hdf5.x on a named nid005829 by dmse643 Fri Nov 26 12:54:34 2021 [0]PETSC ERROR: Configure options --known-has-attribute-aligned=1 --known-mpi-int64_t=0 --known-bits-per-byte=8 --known-64-bit-blas-indices=0 --known-sdot-returns-double=0 --known-snrm2-returns-double=0 --known-level1-dcache-assoc=4 --known-level1-dcache-linesize=64 --known-level1-dcache-size=16384 --known-memcmp-ok=1 --known-mpi-c-double-complex=1 --known-mpi-long-double=1 --known-mpi-shared-libraries=0 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8 --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8 --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8 --known-sizeof-void-p=8 --with-ar=ar --with-batch=0 --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 --with-debugging=0 --with-dependencies=0 --with-fc=ftn --with-fortran-datatypes=1 --with-fortran-interfaces=1 --with-fortran-bindings=1 --with-fortranlib-autodetect=0 --with-ranlib=ranlib --with-scalar-type=real --with-shared-ld=ar --with-etags=0 --with-x=0 --with-ssl=0 --with-shared-libraries=0 --with-mpi-lib="[]" --with-mpi-include="[]" --with-mpiexec=srun --with-blaslapack=1 --with-superlu=1 --with-superlu_dist=1 --with-parmetis=1 --with-metis=1 --with-hypre=1 --with-scalapack=1 --with-ptscotch=1 --with-ptscotch-include= --with-ptscotch-lib= --with-mumps=1 --with-mumps-include= --with-mumps-lib=-lmpifort --with-hdf5=1 --CFLAGS="-O3 -ffast-math -fPIC " --CPPFLAGS="-I/work/y07/shared/libs/core/petsc/3.14.2/GNU/9.3/include " --CXXFLAGS="-O3 -ffast-math -fPIC " --with-cxx-dialect=C++11 --FFLAGS="-O3 -ffast-math -fPIC " --LDFLAGS="-L/work/y07/shared/libs/core/petsc/3.14.2/GNU/9.3/lib " --LIBS="-lgfortran -lgcc -lstdc++" --PETSC_ARCH=x86-rome --prefix=/work/y07/shared/libs/core/petsc/3.14.2/GNU/9.3 [0]PETSC ERROR: #4 PetscViewerFileClose_HDF5() line 79 in /mnt/lustre/a2fs-work1/work/y07/y07/cse/pe-scripts/petsc-3.14.2/src/sys/classes/viewer/impls/hdf5/hdf5v.c [0]PETSC ERROR: #5 PetscViewerDestroy_HDF5() line 90 in /mnt/lustre/a2fs-work1/work/y07/y07/cse/pe-scripts/petsc-3.14.2/src/sys/classes/viewer/impls/hdf5/hdf5v.c Writing 5 [0]PETSC ERROR: #6 PetscViewerDestroy() line 118 in /mnt/lustre/a2fs-work1/work/y07/y07/cse/pe-scripts/petsc-3.14.2/src/sys/classes/viewer/interface/view.c Writing 6 Writing 7 Writing 8 -------------- next part -------------- program hdf5_test_program use petsc implicit none #include "petsc/finclude/petsc.h" integer :: ierr logical :: found DM :: da_w Vec :: w_vec PetscViewer :: viewer integer :: num_procs PetscInt :: global_dim_x, global_dim_y, global_dim_z PetscInt, parameter :: dof = 1 PetscInt, parameter :: stencil_width = 1 PetscInt, dimension(0:0) :: x_lengths PetscInt, dimension(0:0) :: y_lengths PetscInt, dimension(0:0) :: z_lengths_w character(len = 30) :: option_name character(len = 60) :: msg call PetscInitialize(PETSC_NULL_CHARACTER, ierr) call MPI_Comm_size(PETSC_COMM_WORLD, num_procs, ierr) if (num_procs .ne. 1) then write(*, *) 'The number of processes must be 1' call MPI_Abort(PETSC_COMM_WORLD, -1, ierr) end if ! Read command-line options and options file. option_name = '-global_dim_x' call PetscOptionsGetInt(PETSC_NULL_OPTIONS, PETSC_NULL_CHARACTER, option_name, global_dim_x, found, ierr) if (found) then write(msg, *) option_name, '=', global_dim_x call PetscPrintf(PETSC_COMM_WORLD, trim(msg) // NEW_LINE('a'), ierr) else write(msg, *) 'Error:', option_name, 'not found.' call PetscPrintf(PETSC_COMM_WORLD, trim(msg) // NEW_LINE('a'), ierr) end if option_name = '-global_dim_y' call PetscOptionsGetInt(PETSC_NULL_OPTIONS, PETSC_NULL_CHARACTER, option_name, global_dim_y, found, ierr) if (found) then write(msg, *) option_name, '=', global_dim_y call PetscPrintf(PETSC_COMM_WORLD, trim(msg) // NEW_LINE('a'), ierr) else write(msg, *) 'Error:', option_name, 'not found.' call PetscPrintf(PETSC_COMM_WORLD, trim(msg) // NEW_LINE('a'), ierr) end if option_name = '-global_dim_z' call PetscOptionsGetInt(PETSC_NULL_OPTIONS, PETSC_NULL_CHARACTER, option_name, global_dim_z, found, ierr) if (found) then write(msg, *) option_name, '=', global_dim_z call PetscPrintf(PETSC_COMM_WORLD, trim(msg) // NEW_LINE('a'), ierr) if(global_dim_z .le. 1) then write(msg, *) 'global_dim_z must be greater than 1.' call PetscPrintf(PETSC_COMM_WORLD, trim(msg) // NEW_LINE('a'), ierr) end if else write(msg, *) 'Error:', option_name, 'not found.' call PetscPrintf(PETSC_COMM_WORLD, trim(msg) // NEW_LINE('a'), ierr) end if x_lengths = global_dim_x y_lengths = global_dim_y z_lengths_w = global_dim_z+1 call DMDACreate3d(PETSC_COMM_WORLD, & DM_BOUNDARY_PERIODIC, & DM_BOUNDARY_PERIODIC, & DM_BOUNDARY_NONE, & DMDA_STENCIL_BOX, & global_dim_x, global_dim_y, global_dim_z+1, & 1, 1, 1, & dof, stencil_width, & x_lengths, & y_lengths, & z_lengths_w, & da_w, ierr);CHKERRA(ierr) call DMSetUp(da_w, ierr);CHKERRA(ierr) call DMCreateGlobalVector(da_w, w_vec, ierr) call VecZeroEntries(w_vec, ierr) call PetscViewerHDF5Open(PETSC_COMM_WORLD, './w.h5', FILE_MODE_WRITE, viewer, ierr) call PetscViewerPushFormat(viewer, PETSC_VIEWER_HDF5_XDMF, ierr) ! Write the Vec without one extra dimension for BS write(*, *) 'Writing 1' call PetscViewerHDF5SetBaseDimension2(viewer, PETSC_FALSE, ierr) write(*, *) 'Writing 2' call PetscObjectSetName(w_vec, 'w', ierr) write(*, *) 'Writing 3' call VecView(w_vec, viewer, ierr) write(*, *) 'Writing 4' call PetscViewerDestroy(viewer, ierr) write(*, *) 'Writing 5' call VecDestroy(w_vec, ierr) write(*, *) 'Writing 6' call DMDestroy(da_w, ierr);CHKERRA(ierr) write(*, *) 'Writing 7' call PetscFinalize(ierr) write(*, *) 'Writing 8' end program hdf5_test_program From wence at gmx.li Fri Nov 26 09:25:19 2021 From: wence at gmx.li (Lawrence Mitchell) Date: Fri, 26 Nov 2021 15:25:19 +0000 Subject: [petsc-users] Error Writing Large HDF5 File In-Reply-To: References: Message-ID: This is failing setting the chunksize: https://gitlab.com/petsc/petsc/-/blob/main/src/dm/impls/da/gr2.c#L517 It is hard for me to follow this code, but it looks like the chunk is just set to the total extent of the DA (on the process?). This can grow too large for HDF5, which has limits described here https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/ chunks must have fewer than 2^32-1 elements and a max size of 4GB I suspect you?re hitting that limit. I guess this part of the code should be fully refactored to set sensible chunk sizes. That same document suggests 1MB is a good option. Lawrence On Fri, 26 Nov 2021 at 15:10, David Scott wrote: > Hello, > > I am trying to write out HDF5 files. The program I have works all right > up to a point but then fails when I double the size of the file that I > am trying to write out. I have attached a file containing the error > message and another file containing the short program that I used to > generate it. > > The first of these files reports that Petsc 3.14.2 was used. I know that > this is old but I have obtained the same result with Petsc 3.16.1. Also, > I have observed the same behaviour on two different machines. I tried > using 64 bit indices but it made no difference. > > The program runs successfully with the following command line options > > -global_dim_x 1344 -global_dim_y 336 -global_dim_z 336 > > but fails with these > > -global_dim_x 2688 -global_dim_y 336 -global_dim_z 336 > > All the best, > > David > > The University of Edinburgh is a charitable body, registered in Scotland, > with registration number SC005336. Is e buidheann carthannais a th? ann an > Oilthigh Dh?n ?ideann, cl?raichte an Alba, ?ireamh cl?raidh SC005336. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Nov 26 09:31:08 2021 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 26 Nov 2021 10:31:08 -0500 Subject: [petsc-users] How to set boundary conditions for a structured finite volume (FV) discretization when using TS/SNES? In-Reply-To: References: Message-ID: <8ABB6088-B147-40BC-BC6D-62438B28FB33@petsc.dev> Could you be more specific in saying how it fails (can you share the actual prototype code?) Note here you seem to be missing a /dx term G[i] = (u[i+1] - u[i]) - q_A; //Neumann BC: du/dx(x=0.0) = q_A; Barry > On Nov 26, 2021, at 4:02 AM, zhfreewill wrote: > > Hi all, > I have a basic question on how to apply B.C. for a cell-centered structured FV discretization when using the SNES and Ts object. > For instance, for a 1-D transient diffusion equation > u_t = u_xx, 0 < x < 1 > with IC: > u(x,t=0) = 0; > and Dirichlet BC: > u(x=0) = u_A = 0.0; > u(x=1) = u_B = 1.0; > or Neumann BC: > du/dx(x=0.0) = q_A = 0.0; > du/dx(x=1.0) = q_B = 1.0; > > When using TS of PETSc, the equation can be organized in form of > U_t = G(t,u) > where G is the right-hand-side (RHS) function regardless of the specific method of discretization. > (1) For FD discretization > > A |---dx--| B > |-------|-------|-------|-------|-------| grid size = nx > 0 i-1 i i+1 nx-1 > > the vertex-centered FD discretization of the domain is > x(i) = i*dx (i = 0,..., nx)), where dx = 1/nx > and the RHS function G of the equation is > G[i] = (u[i+1]-2*u[i]+u[i-1])/dx^2 > > There have been dozens of examples demonstrating way of BC setting with a finite difference medthod, e.g., > https://github.com/petsc/petsc/blob/386ebab930a846575609b49074a40c46e7f8ed75/src/ts/tutorials/ex26.c#L293-L343 > which I think I have understood how it works and the following lines of code evaluate the RHS function G(t,u) in PETSc's TSSetRHSFunction() routine > /*------------------FD discretization-------------------------*/ > for (i = xs; i < xs + xm; i++) { > // left boundary > if (i == 0) { > G[i] = u[i] - u_A; // Dirichlet BC: u(x=0) = u_A; > G[i] = (u[i] - u[i+1])/dx - q_A; // Neumann BC: du/dx(x=0.0) = q_A; > } > // right boundary > else if (i == mx - 1) { > G[i] = u[i] - u_B; // Dirichlet BC: u(x=1.0) = u_B; > G[i] = (u[i]-u[i-1])/dx - q_B; // Neumann BC: du/dx(x=0.0) = q_A; > } > // internal > else { > G[i] = (u[i-1] - 2 * u[i] + u[i + 1])/dx^2; //u_xx > } > }/*------------------------------------------------------------*/ > > The BCs, e.g., at the left boudnary, can be straightforwardly set as follows: > Dirichlet BC: > G[0] = u[0] - u_A > which indicates G[0] = u[0] - u_A = 0 and Dirichlet BC u[0] = u_A is thus applied > Neumann BC: > G[0] = (u[0] - u[1])/dx - q_A > which indicates G[0] = (u[0] - u[1])/dx - q_A = 0 and Neumann BC du/dx = (u[0] - u[1])/dx = q_A is applied > This is the way the above examples apply BCs. > > For a structured, cell-centered FVM discretization, however, I tried and found the BCs can not be applied in a similar way. I will explain how I tried this as follows. > (2) FV discretization > A |---dx--| B > |---o---|---o---|---o---|---o---|---o---| cell size = nx > W w P e E > 0 i-1 i i+1 nx-1 > > (NOTE: the center cell is denoted as P and its neighbouring cells as W, E; cell faces are denoted as w, e; the left and right boundary faces are A and B, respectively.) > > The cell-centered FV discretization of the domain is > x(i) = dx/2 + i*dx (i = 0,..., nx-1)), where dx = 1/nx > the diffusion term u_xx can be reorganized by using divergence theorem > (u_E-u_P) (u_P-u_W) > --------- - --------- > dx dx > and the RHS function G of the equation is > G[i] = (u_E-u_P)/dx - (u_P-u_W)/dx = (u[i+1]-2*u[i]+u[i-1])/dx > the key lines of code in RHS function G(t,u) in PETSc's TSSetRHSFunction() routine can be > > /*------------------FV discretization-------------------------*/ > for (i = xs; i < xs + xm; i++) { > // left boundary > if (i == 0) { > G[i] = 3.0 * u[i] - u[i+1] - 2.0 * u_A; //Dirichlet BC: u(x=0.0) = u_A, does not work > G[i] = (u[i+1] - u[i]) - q_A; //Neumann BC: du/dx(x=0.0) = q_A; > } > // right boundary > else if (i == mx - 1) { > G[i] = 3.0 * u[i] - u[i-1] - 2.0 * u_B; //Dirichlet BC: u(x=1.0) = u_B, does not work > G[i] = (u[i]-u[i-1])/dx - q_B; //Neumann BC: du/dx(x=1.0) = q_B, does not work > } > // internal > else { > G[i] = (u[i-1] - 2*u[i] + u[i+1])/dx; //u_x > } > }/*------------------------------------------------------------*/ > > I tried to apply BCs in a way similar to that of the FD discretization. > For the Dirichlet BC on the left boundary, because of the cell-centered arrangement of u[i], the left BC is given as u_A at cell face A instead of cell center. > So, I first applied a linear interpolation to get the cell-centered value of u[0] from the left boundary value u_A and neighbouring u[1] > > u[0] = (u[1]+2*u_A)/3.0 (equivalent to 3*u[0] - u[1] - 2*u_A = 0.0) > > |-dx/2-|------dx-----| > |------o------|------o------| linear interpolation: u[0] = (u[1]+2*u_A)/3.0 > A P e E > u_A u[0] u[1] > > Dirichlet BC: > G[0] = 3.0*u[0] - u[1] - 2.0*u_A > in which u[0] = (u[1]+2*u_A)/3.0 is implicitly calculated and I expect the Dirichlet BC u(x=0) = u_A is applied > > For the Neumann BC, left BC is given as a gradient of u, q_A, at cell face A > Neumann BC: > G[i] = (u[1] - u[0])/dx - q_A > in which Neumann BC du/dx = (u[1] - u[0])/dx = q_A is expected to be applied. > > However, this FV discretization failed to work. Is there anything wrong or could you give me a hint? > > Best, > Qw -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Nov 26 09:34:45 2021 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 26 Nov 2021 10:34:45 -0500 Subject: [petsc-users] DMPlex shared node indexing In-Reply-To: References: Message-ID: Actually, with DMPLEX the ghost vertices are not at the end of the local vector, they can be anywhere within the list. But I have a work-in-progress branch https://gitlab.com/petsc/petsc/-/merge_requests/4542 that labels the ghost locations and moves them to the end. You are welcome to try it. Barry > On Nov 26, 2021, at 8:16 AM, Mark Adams wrote: > > First, ghost vertices only show up in "local" (vs "global") vectors and the ghost values are ordered after the locally owned ones. > Others might have more to add. > Mark > > > On Fri, Nov 26, 2021 at 12:49 AM ?? > wrote: > Dear PETSc-team, > > When the mesh is distributed, the vertices are renumbered and some vertices are shared by neighboring CPUs. My question is: > > 1. How to check which vertice is a ghost one? > 2. Are those ghost vertices always at the end of vertices list? If not > 3. Are there simple ways to let those ghost vertices come after owned ones? > > Many thanks > > Yuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsmith at petsc.dev Fri Nov 26 10:56:17 2021 From: bsmith at petsc.dev (Barry Smith) Date: Fri, 26 Nov 2021 11:56:17 -0500 Subject: [petsc-users] How to check NULL pointer in Fortran In-Reply-To: References: <5CCCE137-D42C-4F1D-8690-195A53504DED@petsc.dev> <9D5E5549-9453-40F7-9F7D-8F71A821485E@petsc.dev> Message-ID: <88620574-B084-41D5-AF73-47CEB0520437@petsc.dev> I believe I have fixed the bug in the branch barry/2021-11-26/fix-petscsfgetgraph-fortran/release https://gitlab.com/petsc/petsc/-/merge_requests/4605 See src/vec/is/sf/tutorials/ex1f.F90 for how to check if the returned value is a FORTRAN_NULL_INTEGER Barry > On Nov 24, 2021, at 3:19 AM, ?? wrote: > > Thank you for the clarification. I am looking forward to this bug fix. > > Yuan > > 2021?11?22?(?) 23:17 Barry Smith >: > > The address of PETSC_NULL_INTEGER is not 0, it in some common block location. PETSC_NULL_INTEGER should not be used in the form > if idx == PETSC_NULL_INTEGER, it is only to be used as an argument to a PETSc function that on the C side can take a NULL (integer) argument. > > As I noted before whoever wrote the incorrect petscsfgetgraph_ Fortran wrapper needs to fix it. > > Barry > > >> On Nov 21, 2021, at 8:05 PM, ?? > wrote: >> >> Thank you very much. >> >> The NULL pointer could be identified by if( loc(gmine)==PETSC_NULL_INTEGER) as above @Mark Adams suggested. However, It seems that there is a memory leak here? >> >> Yuan >> >> 2021?11?19?(?) 22:20 Barry Smith >: >> >> it is very possible there is a bug in the Fortran interface for this function. It looks like whoever wrote the Fortran interface did not think through the special case of contiguous entries. >> >> PETSC_EXTERN void petscsfgetgraph_(PetscSF *sf,PetscInt *nroots,PetscInt *nleaves, F90Array1d *ailocal, F90Array1d *airemote, PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(pilocal) PETSC_F90_2PTR_PROTO(piremote)) >> { >> const PetscInt *ilocal; >> const PetscSFNode *iremote; >> >> *ierr = PetscSFGetGraph(*sf,nroots,nleaves,&ilocal,&iremote);if (*ierr) return; >> *ierr = F90Array1dCreate((void*)ilocal,MPIU_INT,1,*nleaves, ailocal PETSC_F90_2PTR_PARAM(pilocal)); >> /* this creates a memory leak */ >> f90arraysfnodecreate_((PetscInt*)iremote,nleaves, airemote PETSC_F90_2PTR_PARAM(piremote)); >> } >> >> PetscErrorCode F90Array1dCreate(void *array,MPI_Datatype type,PetscInt start,PetscInt len,F90Array1d *ptr PETSC_F90_2PTR_PROTO(ptrd)) >> { >> PetscFunctionBegin; >> if (type == MPIU_SCALAR) { >> if (!len) array = PETSC_NULL_SCALAR_Fortran; >> f90array1dcreatescalar_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); >> } else if (type == MPIU_REAL) { >> if (!len) array = PETSC_NULL_REAL_Fortran; >> f90array1dcreatereal_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); >> } else if (type == MPIU_INT) { >> if (!len) array = PETSC_NULL_INTEGER_Fortran; >> f90array1dcreateint_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); >> } else if (type == MPIU_FORTRANADDR) { >> f90array1dcreatefortranaddr_(array,&start,&len,ptr PETSC_F90_2PTR_PARAM(ptrd)); >> } else SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Unsupported MPI_Datatype"); >> PetscFunctionReturn(0); >> } >> >> It blindly calls F90Array1dCreate() with a len = *nleaves > 0 but a NULL array ilocal then f90array1dcreateint_ must create an invalid object. >> >> Barry >> >> >> >> > On Nov 19, 2021, at 12:34 AM, ?? > wrote: >> > >> > Dear PETSc-team, >> > >> > I am using function PetscSFGetGraph in my program like >> > ----------------------- >> > call PetscSFGetGraph(sf,gnroots,gnleaves,gmine,gremote,ierr) >> > ----------------------- >> > >> > In some cases, it works well. But in some cases, I encountered following error >> > ------------------ >> > PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >> > ------------------ >> > I found it is due to the action of reading gmine, which is a fortran pointer point to an array. It is reasonable because PETSc manual tells me "if returned value is NULL, it means leaves are in contiguous storage". The problem is that I cannot find if gmine is a null pointer. >> > 1) I cannot use "if (gmine==PETSC_NULL_INTEGER)" because my intel compiler would return a compile error "A scalar-valued expression is required in this context" >> > 2) When using standard style of checking a null pointer in fortran, "associated(gmine)", it returns "T". Even in cases such action of "print *, gmine(1)" would give rise to above Segmentation Violation error. >> > Is there any means to check the NULL pointer in Fortran in above cases. >> > Many thanks, >> > Yuan >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rossi.f at inwind.it Fri Nov 26 11:40:10 2021 From: rossi.f at inwind.it (Fabio Rossi) Date: Fri, 26 Nov 2021 18:40:10 +0100 (CET) Subject: [petsc-users] configure error with fftw and without mpi In-Reply-To: References: <400981621.484717.1637835303203@mail1.libero.it> Message-ID: <421099383.568384.1637948410768@mail1.libero.it> Thank you Barry! Backported the patch to 3.16.0 and tested! I'll try to integrate downstream in Gentoo (https://bugs.gentoo.org/827185). Fabio > Il 26/11/2021 05:15 Barry Smith ha scritto: > > > > I have made a branch?barry/2021-11-25/feature-fftw-seq?https://gitlab.com/petsc/petsc/-/merge_requests/4604 that provides support for using FFTW without MPI, please let us know if you have difficulties with it. > > > Barry > > > > > > On Nov 25, 2021, at 5:15 AM, Fabio Rossi via petsc-users wrote: > > > > > > I am trying to compile petsc with the following options (Gentoo build system): > > > > ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libdir=/usr/lib64/petsc/lib64 scrollOutput=1 FFLAGS=-O2 -pipe -fPIC CFLAGS=-march=core2 -O2 -pipe -fPIC CXXFLAGS=-march=core2 -O2 -pipe -fPIC LDFLAGS=-Wl,-O1 -Wl,--as-needed --prefix=/usr/lib64/petsc --with-shared-libraries --with-single-library --with-clanguage=c --with-petsc-arch=linux-gnu-c-opt --with-precision=double --with-gnu-compilers --with-blas-lapack-lib=-lblas -llapack --with-debugging=0 --with-mpi=0 --with-cc=x86_64-pc-linux-gnu-gcc --with-cxx=x86_64-pc-linux-gnu-g++ --with-fortran=1 --with-fc=x86_64-pc-linux-gnu-gfortran --with-mpi-compilers=0 --with-scalar-type=real --with-windows-graphics=0 --with-matlab=0 --with-cmake:BOOL=1 --with-pthread=1 --with-afterimage=0 --without-hdf5 --with-hypre=0 --without-suitesparse --with-superlu=0 --with-x --with-x11 --with-ptscotch=0 > > --with- > > scalapack=0 --without-mumps --with-imagemagick=0 --with-python=0 --with-boost --with-fftw > > > > so MPI support is disabled, and I get the following configure error: > > > > TESTING: consistencyChecks from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:963) > > TESTING: checkDependencies from config.packages.fftw(/var/tmp/portage/sci-mathematics/petsc-3.15.0/work/petsc-3.15.0/config/BuildSystem/config/package.py:872) > > ******************************************************************************* > > UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for details): > > ------------------------------------------------------------------------------- > > Did not find package MPI needed by fftw. > > Enable the package using --with-mpi > > > > Isn't MPI optional? Where is the error coming from? > > > > If I disable fftw then it compiles successfully. > > > > Fabio > From knepley at gmail.com Fri Nov 26 14:04:31 2021 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Nov 2021 15:04:31 -0500 Subject: [petsc-users] How to set boundary conditions for a structured finite volume (FV) discretization when using TS/SNES? In-Reply-To: <8ABB6088-B147-40BC-BC6D-62438B28FB33@petsc.dev> References: <8ABB6088-B147-40BC-BC6D-62438B28FB33@petsc.dev> Message-ID: What we do in the FVM examples is to use ghost cells around the boundary. Then the state in these cells is set so that the flux on the boundary surface is exactly what you want. You could also prescribe the fluxes directly, but that is not as natural in the setup I have used for FVM. Thanks, Matt On Fri, Nov 26, 2021 at 10:31 AM Barry Smith wrote: > > Could you be more specific in saying how it fails (can you share the > actual prototype code?) > > Note here you seem to be missing a /dx term > > *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) = > q_A;* > > > Barry > > On Nov 26, 2021, at 4:02 AM, zhfreewill wrote: > > Hi all, > I have a basic question on how to apply B.C. for a cell-centered > structured FV discretization when using the SNES and Ts object. > For instance, for a 1-D transient diffusion equation > > *u_t = u_xx, 0 < x < 1* > > with IC: > > *u(x,t=0) = 0;* > > and Dirichlet BC: > > *u(x=0) = u_A = 0.0;* > *u(x=1) = u_B = 1.0;* > > or Neumann BC: > > *du/dx(x=0.0) = q_A = 0.0;* > *du/dx(x=1.0) = q_B = 1.0;* > > > When using TS of PETSc, the equation can be organized in form of > *U_t = G(t,u)* > where *G* is the right-hand-side (RHS) function regardless of the > specific method of discretization. > *(1) For FD discretization* > > *A |---dx--| B* > *|-------|-------|-------|-------|-------| grid size = nx* > *0 i-1 i i+1 nx-1* > > the vertex-centered FD discretization of the domain is > > *x(i) = i*dx (i = 0,..., nx)), where dx = 1/nx* > > and the RHS function G of the equation is > > *G[i] = (u[i+1]-2*u[i]+u[i-1])/dx^2* > > There have been dozens of examples demonstrating way of BC setting with a > finite difference medthod, e.g., > > https://github.com/petsc/petsc/blob/386ebab930a846575609b49074a40c46e7f8ed75/src/ts/tutorials/ex26.c#L293-L343 > which I think I have understood how it works and the following lines of > code evaluate the RHS function *G(t,u)* in PETSc's *TSSetRHSFunction()* > routine > /*------------------FD discretization-------------------------*/ > *for (i = xs; i < xs + xm; i++) {* > > *// left boundary* > *if (i == 0) {* > > *G[i] = u[i] - u_A; **// Dirichlet BC: u(x=0) = u_A;* > *G[i] = (u[i] - u[i+1])/dx - q_A; **// Neumann BC: du/dx(x=0.0) = q_A;* > > *} * > *// right boundary* > *else if (i == mx - 1) {* > > *G[i] = u[i] - u_B; **// Dirichlet BC: u(x=1.0) = u_B;* > *G[i] = (u[i]-u[i-1])/dx - q_B; **// Neumann BC: du/dx(x=0.0) = q_A;* > > *}* > *// internal* > *else {* > > *G[i] = (u[i-1] - 2 * u[i] + u[i + 1])/dx^2; **//u_xx* > > *}* > > *}**/*------------------------------------------------------------*/* > > The BCs, e.g., at the left boudnary, can be straightforwardly set as > follows: > Dirichlet BC: > *G[0] = u[0] - u_A* > which indicates *G[0] = u[0] - u_A = 0* and Dirichlet BC *u[0] = u_A* is > thus applied > Neumann BC: > *G[0] = (u[0] - u[1])/dx - q_A* > which indicates *G[0] = (u[0] - u[1])/dx - q_A = 0* and Neumann BC *du/dx > = (u[0] - u[1])/dx = q_A* is applied > This is the way the above examples apply BCs. > > For a structured, cell-centered FVM discretization, however, I tried and > found the BCs can not be applied in a similar way. I will explain how I > tried this as follows. > *(2) FV discretization* > > *A |---dx--| B* > *|---o---|---o---|---o---|---o---|---o---| cell size = nx* > *W w P e E * > *0 i-1 i i+1 nx-1* > > (NOTE: the center cell is denoted as P and its neighbouring cells as W, E; cell > faces are denoted as w, e; the left and right boundary faces are A and B, > respectively.) > > The cell-centered FV discretization of the domain is > > *x(i) = dx/2 + i*dx (i = 0,..., nx-1)), where dx = 1/nx* > > the diffusion term u_xx can be reorganized by using divergence theorem > > *(u_E-u_P) (u_P-u_W)* > *--------- - ---------* > * dx dx* > > and the RHS function G of the equation is > > *G[i] = (u_E-u_P)/dx - (u_P-u_W)/dx = (u[i+1]-2*u[i]+u[i-1])/dx* > > the key lines of code in RHS function G(t,u) in PETSc's > *TSSetRHSFunction()* routine can be > > */*------------------FV discretization-------------------------*/* > *for (i = xs; i < xs + xm; i++) {* > > *// left boundary* > *if (i == 0) {* > > *G[i] = 3.0 * u[i] - u[i+1] - 2.0 * u_A; **//Dirichlet BC: u(x=0.0) = > u_A, does not work* > *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) = > q_A;* > > *}* > *// right boundary* > *else if (i == mx - 1) {* > > *G[i] = 3.0 * u[i] - u[i-1] - 2.0 * u_B; **//Dirichlet BC: u(x=1.0) = > u_B, does not work* > *G[i] = (u[i]-u[i-1])/dx - q_B; **//Neumann BC: du/dx(x=1.0) = > q_B, does not work* > > *}* > *// internal* > *else {* > > *G[i] = (u[i-1] - 2*u[i] + u[i+1])/dx; **//u_x* > > *}* > > *}**/*------------------------------------------------------------*/* > > I tried to apply BCs in a way similar to that of the FD discretization. > For the Dirichlet BC on the left boundary, because of the cell-centered > arrangement of u[i], the left BC is given as *u_A* at cell face A instead > of cell center. > So, I first applied a linear interpolation to get the cell-centered value > of *u[0]* from the left boundary value *u_A* and neighbouring *u[1]* > > *u[0] = (u[1]+2*u_A)/3.0 (equivalent to 3*u[0] - u[1] - 2*u_A = 0.0)* > > * |-dx/2-|------dx-----|* > * |------o------|------o------| linear interpolation: u[0] = > (u[1]+2*u_A)/3.0* > * A P e E* > * u_A u[0] u[1]* > > Dirichlet BC: > *G[0] = 3.0*u[0] - u[1] - 2.0*u_A* > in which *u[0] = (u[1]+2*u_A)/3.0* is implicitly calculated and I > expect the Dirichlet BC *u(x=0) = u_A* is applied > > For the Neumann BC, left BC is given as a gradient of *u*, *q_A*, at cell > face A > Neumann BC: > *G[i] = (u[1] - u[0])/dx - q_A* > in which Neumann BC *du/dx = (u[1] - u[0])/dx = q_A* is expected to > be applied. > > However, this FV discretization failed to work. Is there anything wrong or > could you give me a hint? > > Best, > Qw > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From knepley at gmail.com Fri Nov 26 14:07:50 2021 From: knepley at gmail.com (Matthew Knepley) Date: Fri, 26 Nov 2021 15:07:50 -0500 Subject: [petsc-users] DMPlex shared node indexing In-Reply-To: References: Message-ID: On Fri, Nov 26, 2021 at 12:49 AM ?? wrote: > Dear PETSc-team, > > When the mesh is distributed, the vertices are renumbered and some > vertices are shared by neighboring CPUs. My question is: > > 1. How to check which vertice is a ghost one? > It is present in the PetscSF describing the connections between local meshes. So you would do DMGetPointSF(dm, &sf); PetscSFGetGraph(sf, NULL, &Nl, &leaves, NULL); If your point in the list 'leaves' of length Nl, then it is a ghost. > 2. Are those ghost vertices always at the end of vertices list? If not > No. > 3. Are there simple ways to let those ghost vertices come after owned ones? > You can introduce a permutation of the points when numbering unknowns. This is what Barry does in his branch. Thanks, Matt > Many thanks > > Yuan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhfreewill at gmail.com Sat Nov 27 01:53:42 2021 From: zhfreewill at gmail.com (zhfreewill) Date: Sat, 27 Nov 2021 15:53:42 +0800 Subject: [petsc-users] How to set boundary conditions for a structured finite volume (FV) discretization when using TS/SNES? In-Reply-To: <8ABB6088-B147-40BC-BC6D-62438B28FB33@petsc.dev> References: <8ABB6088-B147-40BC-BC6D-62438B28FB33@petsc.dev> Message-ID: Hi,Barry, Thanks for reply. I meant I cannot get the right result as the FD version did, although the code did not report any error when running. Indeed, there were mistakes and I fixed it in the code (I also attached additional derivation and code this time, please see the attachment). Still, I cannot get the right output so far. Regards, Qw Barry Smith ?2021?11?26??? ??11:31??? > > Could you be more specific in saying how it fails (can you share the > actual prototype code?) > > Note here you seem to be missing a /dx term > > *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) = > q_A;* > > > Barry > > On Nov 26, 2021, at 4:02 AM, zhfreewill wrote: > > Hi all, > I have a basic question on how to apply B.C. for a cell-centered > structured FV discretization when using the SNES and Ts object. > For instance, for a 1-D transient diffusion equation > > *u_t = u_xx, 0 < x < 1* > > with IC: > > *u(x,t=0) = 0;* > > and Dirichlet BC: > > *u(x=0) = u_A = 0.0;* > *u(x=1) = u_B = 1.0;* > > or Neumann BC: > > *du/dx(x=0.0) = q_A = 0.0;* > *du/dx(x=1.0) = q_B = 1.0;* > > > When using TS of PETSc, the equation can be organized in form of > *U_t = G(t,u)* > where *G* is the right-hand-side (RHS) function regardless of the > specific method of discretization. > *(1) For FD discretization* > > *A |---dx--| B* > *|-------|-------|-------|-------|-------| grid size = nx* > *0 i-1 i i+1 nx-1* > > the vertex-centered FD discretization of the domain is > > *x(i) = i*dx (i = 0,..., nx)), where dx = 1/nx* > > and the RHS function G of the equation is > > *G[i] = (u[i+1]-2*u[i]+u[i-1])/dx^2* > > There have been dozens of examples demonstrating way of BC setting with a > finite difference medthod, e.g., > > https://github.com/petsc/petsc/blob/386ebab930a846575609b49074a40c46e7f8ed75/src/ts/tutorials/ex26.c#L293-L343 > which I think I have understood how it works and the following lines of > code evaluate the RHS function *G(t,u)* in PETSc's *TSSetRHSFunction()* > routine > /*------------------FD discretization-------------------------*/ > *for (i = xs; i < xs + xm; i++) {* > > *// left boundary* > *if (i == 0) {* > > *G[i] = u[i] - u_A; **// Dirichlet BC: u(x=0) = u_A;* > *G[i] = (u[i] - u[i+1])/dx - q_A; **// Neumann BC: du/dx(x=0.0) = q_A;* > > *} * > *// right boundary* > *else if (i == mx - 1) {* > > *G[i] = u[i] - u_B; **// Dirichlet BC: u(x=1.0) = u_B;* > *G[i] = (u[i]-u[i-1])/dx - q_B; **// Neumann BC: du/dx(x=0.0) = q_A;* > > *}* > *// internal* > *else {* > > *G[i] = (u[i-1] - 2 * u[i] + u[i + 1])/dx^2; **//u_xx* > > *}* > > *}**/*------------------------------------------------------------*/* > > The BCs, e.g., at the left boudnary, can be straightforwardly set as > follows: > Dirichlet BC: > *G[0] = u[0] - u_A* > which indicates *G[0] = u[0] - u_A = 0* and Dirichlet BC *u[0] = u_A* is > thus applied > Neumann BC: > *G[0] = (u[0] - u[1])/dx - q_A* > which indicates *G[0] = (u[0] - u[1])/dx - q_A = 0* and Neumann BC *du/dx > = (u[0] - u[1])/dx = q_A* is applied > This is the way the above examples apply BCs. > > For a structured, cell-centered FVM discretization, however, I tried and > found the BCs can not be applied in a similar way. I will explain how I > tried this as follows. > *(2) FV discretization* > > *A |---dx--| B* > *|---o---|---o---|---o---|---o---|---o---| cell size = nx* > *W w P e E * > *0 i-1 i i+1 nx-1* > > (NOTE: the center cell is denoted as P and its neighbouring cells as W, E; cell > faces are denoted as w, e; the left and right boundary faces are A and B, > respectively.) > > The cell-centered FV discretization of the domain is > > *x(i) = dx/2 + i*dx (i = 0,..., nx-1)), where dx = 1/nx* > > the diffusion term u_xx can be reorganized by using divergence theorem > > *(u_E-u_P) (u_P-u_W)* > *--------- - ---------* > * dx dx* > > and the RHS function G of the equation is > > *G[i] = (u_E-u_P)/dx - (u_P-u_W)/dx = (u[i+1]-2*u[i]+u[i-1])/dx* > > the key lines of code in RHS function G(t,u) in PETSc's > *TSSetRHSFunction()* routine can be > > */*------------------FV discretization-------------------------*/* > *for (i = xs; i < xs + xm; i++) {* > > *// left boundary* > *if (i == 0) {* > > *G[i] = 3.0 * u[i] - u[i+1] - 2.0 * u_A; **//Dirichlet BC: u(x=0.0) = > u_A, does not work* > *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) = > q_A;* > > *}* > *// right boundary* > *else if (i == mx - 1) {* > > *G[i] = 3.0 * u[i] - u[i-1] - 2.0 * u_B; **//Dirichlet BC: u(x=1.0) = > u_B, does not work* > *G[i] = (u[i]-u[i-1])/dx - q_B; **//Neumann BC: du/dx(x=1.0) = > q_B, does not work* > > *}* > *// internal* > *else {* > > *G[i] = (u[i-1] - 2*u[i] + u[i+1])/dx; **//u_x* > > *}* > > *}**/*------------------------------------------------------------*/* > > I tried to apply BCs in a way similar to that of the FD discretization. > For the Dirichlet BC on the left boundary, because of the cell-centered > arrangement of u[i], the left BC is given as *u_A* at cell face A instead > of cell center. > So, I first applied a linear interpolation to get the cell-centered value > of *u[0]* from the left boundary value *u_A* and neighbouring *u[1]* > > *u[0] = (u[1]+2*u_A)/3.0 (equivalent to 3*u[0] - u[1] - 2*u_A = 0.0)* > > * |-dx/2-|------dx-----|* > * |------o------|------o------| linear interpolation: u[0] = > (u[1]+2*u_A)/3.0* > * A P e E* > * u_A u[0] u[1]* > > Dirichlet BC: > *G[0] = 3.0*u[0] - u[1] - 2.0*u_A* > in which *u[0] = (u[1]+2*u_A)/3.0* is implicitly calculated and I > expect the Dirichlet BC *u(x=0) = u_A* is applied > > For the Neumann BC, left BC is given as a gradient of *u*, *q_A*, at cell > face A > Neumann BC: > *G[i] = (u[1] - u[0])/dx - q_A* > in which Neumann BC *du/dx = (u[1] - u[0])/dx = q_A* is expected to > be applied. > > However, this FV discretization failed to work. Is there anything wrong or > could you give me a hint? > > Best, > Qw > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: UnsteadyDiffusion1D.c Type: application/octet-stream Size: 6555 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Derivation.zip Type: application/x-zip-compressed Size: 2607803 bytes Desc: not available URL: From bsmith at petsc.dev Sat Nov 27 10:17:45 2021 From: bsmith at petsc.dev (Barry Smith) Date: Sat, 27 Nov 2021 11:17:45 -0500 Subject: [petsc-users] DMPlex shared node indexing In-Reply-To: References: Message-ID: <634BF347-17FA-4FDB-BB7B-42AAFFF9C8B5@petsc.dev> https://gitlab.com/petsc/petsc/-/merge_requests/4542/diffs?commit_id=252cf7895bb78ed4e3a2388b4d15d1e7905925a9 If you use the option -dm_plex_use_vec_ghost_permutation then ghost values will be put at the end of the local vector. Note this is work in progress so will have rough edges. > On Nov 27, 2021, at 10:54 AM, ?? wrote: > > Great! It is of much help to me. But I am not sure how to use it. Does it mean ghost vertices are put at the end of the local vector automatically after calling DMPlexDistribute? Or additional function calling or setting is needed? > > Yuan > > 2021?11?27?(?) 0:34 Barry Smith >: > > Actually, with DMPLEX the ghost vertices are not at the end of the local vector, they can be anywhere within the list. > > But I have a work-in-progress branch https://gitlab.com/petsc/petsc/-/merge_requests/4542 that labels the ghost locations and moves them to the end. You are welcome to try it. > > Barry > > >> On Nov 26, 2021, at 8:16 AM, Mark Adams > wrote: >> >> First, ghost vertices only show up in "local" (vs "global") vectors and the ghost values are ordered after the locally owned ones. >> Others might have more to add. >> Mark >> >> >> On Fri, Nov 26, 2021 at 12:49 AM ?? > wrote: >> Dear PETSc-team, >> >> When the mesh is distributed, the vertices are renumbered and some vertices are shared by neighboring CPUs. My question is: >> >> 1. How to check which vertice is a ghost one? >> 2. Are those ghost vertices always at the end of vertices list? If not >> 3. Are there simple ways to let those ghost vertices come after owned ones? >> >> Many thanks >> >> Yuan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Sat Nov 27 19:36:44 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Sun, 28 Nov 2021 10:36:44 +0900 Subject: [petsc-users] DMPlex shared node indexing In-Reply-To: <634BF347-17FA-4FDB-BB7B-42AAFFF9C8B5@petsc.dev> References: <634BF347-17FA-4FDB-BB7B-42AAFFF9C8B5@petsc.dev> Message-ID: Thanks, I'll check it out. Yuan 2021?11?28?(?) 1:17 Barry Smith : > > > https://gitlab.com/petsc/petsc/-/merge_requests/4542/diffs?commit_id=252cf7895bb78ed4e3a2388b4d15d1e7905925a9 > > If you use the option -dm_plex_use_vec_ghost_permutation then ghost > values will be put at the end of the local vector. Note this is work in > progress so will have rough edges. > > > On Nov 27, 2021, at 10:54 AM, ?? wrote: > > Great! It is of much help to me. But I am not sure how to use it. Does it > mean ghost vertices are put at the end of the local vector automatically > after calling DMPlexDistribute? Or additional function calling or setting > is needed? > > Yuan > > 2021?11?27?(?) 0:34 Barry Smith : > >> >> Actually, with DMPLEX the ghost vertices are not at the end of the >> local vector, they can be anywhere within the list. >> >> But I have a work-in-progress branch >> https://gitlab.com/petsc/petsc/-/merge_requests/4542 that labels the >> ghost locations and moves them to the end. You are welcome to try it. >> >> Barry >> >> >> On Nov 26, 2021, at 8:16 AM, Mark Adams wrote: >> >> First, ghost vertices only show up in "local" (vs "global") vectors and >> the ghost values are ordered after the locally owned ones. >> Others might have more to add. >> Mark >> >> >> On Fri, Nov 26, 2021 at 12:49 AM ?? wrote: >> >>> Dear PETSc-team, >>> >>> When the mesh is distributed, the vertices are renumbered and some >>> vertices are shared by neighboring CPUs. My question is: >>> >>> 1. How to check which vertice is a ghost one? >>> 2. Are those ghost vertices always at the end of vertices list? If not >>> 3. Are there simple ways to let those ghost vertices come after >>> owned ones? >>> >>> Many thanks >>> >>> Yuan >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Sun Nov 28 06:40:33 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Sun, 28 Nov 2021 21:40:33 +0900 Subject: [petsc-users] How to check NULL pointer in Fortran In-Reply-To: <88620574-B084-41D5-AF73-47CEB0520437@petsc.dev> References: <5CCCE137-D42C-4F1D-8690-195A53504DED@petsc.dev> <9D5E5549-9453-40F7-9F7D-8F71A821485E@petsc.dev> <88620574-B084-41D5-AF73-47CEB0520437@petsc.dev> Message-ID: Awesome, thanks! I'll check it out. Yuan 2021?11?27?(?) 1:56 Barry Smith : > > I believe I have fixed the bug in the branch > *barry/2021-11-26/fix-petscsfgetgraph-fortran/release * > https://gitlab.com/petsc/petsc/-/merge_requests/4605 > > See src/vec/is/sf/tutorials/ex1f.F90 for how to check if the returned > value is a FORTRAN_NULL_INTEGER > > Barry > > > On Nov 24, 2021, at 3:19 AM, ?? wrote: > > Thank you for the clarification. I am looking forward to this bug fix. > > Yuan > > 2021?11?22?(?) 23:17 Barry Smith : > >> >> The address of PETSC_NULL_INTEGER is not 0, it in some common block >> location. PETSC_NULL_INTEGER should not be used in the form >> if idx == PETSC_NULL_INTEGER, it is only to be used as an argument to a >> PETSc function that on the C side can take a NULL (integer) argument. >> >> As I noted before whoever wrote the incorrect petscsfgetgraph_ Fortran >> wrapper needs to fix it. >> >> Barry >> >> >> On Nov 21, 2021, at 8:05 PM, ?? wrote: >> >> Thank you very much. >> >> The NULL pointer could be identified by if( >> loc(gmine)==PETSC_NULL_INTEGER) as above @Mark Adams suggested. >> However, It seems that there is a memory leak here? >> >> Yuan >> >> 2021?11?19?(?) 22:20 Barry Smith : >> >>> >>> it is very possible there is a bug in the Fortran interface for this >>> function. It looks like whoever wrote the Fortran interface did not think >>> through the special case of contiguous entries. >>> >>> PETSC_EXTERN void petscsfgetgraph_(PetscSF *sf,PetscInt >>> *nroots,PetscInt *nleaves, F90Array1d *ailocal, F90Array1d *airemote, >>> PetscErrorCode *ierr PETSC_F90_2PTR_PROTO(pilocal) >>> PETSC_F90_2PTR_PROTO(piremote)) >>> { >>> const PetscInt *ilocal; >>> const PetscSFNode *iremote; >>> >>> *ierr = PetscSFGetGraph(*sf,nroots,nleaves,&ilocal,&iremote);if >>> (*ierr) return; >>> *ierr = F90Array1dCreate((void*)ilocal,MPIU_INT,1,*nleaves, ailocal >>> PETSC_F90_2PTR_PARAM(pilocal)); >>> /* this creates a memory leak */ >>> f90arraysfnodecreate_((PetscInt*)iremote,nleaves, airemote >>> PETSC_F90_2PTR_PARAM(piremote)); >>> } >>> >>> PetscErrorCode F90Array1dCreate(void *array,MPI_Datatype type,PetscInt >>> start,PetscInt len,F90Array1d *ptr PETSC_F90_2PTR_PROTO(ptrd)) >>> { >>> PetscFunctionBegin; >>> if (type == MPIU_SCALAR) { >>> if (!len) array = PETSC_NULL_SCALAR_Fortran; >>> f90array1dcreatescalar_(array,&start,&len,ptr >>> PETSC_F90_2PTR_PARAM(ptrd)); >>> } else if (type == MPIU_REAL) { >>> if (!len) array = PETSC_NULL_REAL_Fortran; >>> f90array1dcreatereal_(array,&start,&len,ptr >>> PETSC_F90_2PTR_PARAM(ptrd)); >>> } else if (type == MPIU_INT) { >>> if (!len) array = PETSC_NULL_INTEGER_Fortran; >>> f90array1dcreateint_(array,&start,&len,ptr >>> PETSC_F90_2PTR_PARAM(ptrd)); >>> } else if (type == MPIU_FORTRANADDR) { >>> f90array1dcreatefortranaddr_(array,&start,&len,ptr >>> PETSC_F90_2PTR_PARAM(ptrd)); >>> } else SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Unsupported >>> MPI_Datatype"); >>> PetscFunctionReturn(0); >>> } >>> >>> It blindly calls F90Array1dCreate() with a len = *nleaves > 0 but a NULL >>> array ilocal then f90array1dcreateint_ must create an invalid object. >>> >>> Barry >>> >>> >>> >>> > On Nov 19, 2021, at 12:34 AM, ?? wrote: >>> > >>> > Dear PETSc-team, >>> > >>> > I am using function PetscSFGetGraph in my program like >>> > ----------------------- >>> > call PetscSFGetGraph(sf,gnroots,gnleaves,gmine,gremote,ierr) >>> > ----------------------- >>> > >>> > In some cases, it works well. But in some cases, I encountered >>> following error >>> > ------------------ >>> > PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> probably memory access out of range >>> > ------------------ >>> > I found it is due to the action of reading gmine, which is a fortran >>> pointer point to an array. It is reasonable because PETSc manual tells me >>> "if returned value is NULL, it means leaves are in contiguous storage". The >>> problem is that I cannot find if gmine is a null pointer. >>> > 1) I cannot use "if (gmine==PETSC_NULL_INTEGER)" because my intel >>> compiler would return a compile error "A scalar-valued expression is >>> required in this context" >>> > 2) When using standard style of checking a null pointer in fortran, >>> "associated(gmine)", it returns "T". Even in cases such action of "print *, >>> gmine(1)" would give rise to above Segmentation Violation error. >>> > Is there any means to check the NULL pointer in Fortran in above cases. >>> > Many thanks, >>> > Yuan >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Sun Nov 28 07:04:43 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Sun, 28 Nov 2021 22:04:43 +0900 Subject: [petsc-users] PETSc Ordering for vectors generated by DMCreateGlobalVector Message-ID: Hi all, Application Ordering of a vector is generally different from its PETSc one. In my application, I wish to retrieve the value of a vector generated by DMCreateGlobalVector into its original application order. I know there are some functions like DMDAGetAO to do this. My question is : Is there any DMPlex version of such a function? If not, I wish to know how to do it. Many thanks Yuan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Sun Nov 28 08:17:41 2021 From: mfadams at lbl.gov (Mark Adams) Date: Sun, 28 Nov 2021 09:17:41 -0500 Subject: [petsc-users] How to set boundary conditions for a structured finite volume (FV) discretization when using TS/SNES? In-Reply-To: References: <8ABB6088-B147-40BC-BC6D-62438B28FB33@petsc.dev> Message-ID: * I have used Matt's approach and it works but you do have to remember to update the ghost cells after you update your solution and before you use the operator again. * I would use a convergence test to determine if the result is correct. * Check that a constant function is a (trivial) solution to your discretization. On Sat, Nov 27, 2021 at 2:54 AM zhfreewill wrote: > Hi,Barry, > Thanks for reply. I meant I cannot get the right result as the FD version > did, although the code did not report any error when running. > > Indeed, there were mistakes and I fixed it in the code (I also attached > additional derivation and code this time, please see the attachment). > Still, I cannot get the right output so far. > > Regards, > Qw > > Barry Smith ?2021?11?26??? ??11:31??? > >> >> Could you be more specific in saying how it fails (can you share the >> actual prototype code?) >> >> Note here you seem to be missing a /dx term >> >> *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) >> = q_A;* >> >> >> Barry >> >> On Nov 26, 2021, at 4:02 AM, zhfreewill wrote: >> >> Hi all, >> I have a basic question on how to apply B.C. for a cell-centered >> structured FV discretization when using the SNES and Ts object. >> For instance, for a 1-D transient diffusion equation >> >> *u_t = u_xx, 0 < x < 1* >> >> with IC: >> >> *u(x,t=0) = 0;* >> >> and Dirichlet BC: >> >> *u(x=0) = u_A = 0.0;* >> *u(x=1) = u_B = 1.0;* >> >> or Neumann BC: >> >> *du/dx(x=0.0) = q_A = 0.0;* >> *du/dx(x=1.0) = q_B = 1.0;* >> >> >> When using TS of PETSc, the equation can be organized in form of >> *U_t = G(t,u)* >> where *G* is the right-hand-side (RHS) function regardless of the >> specific method of discretization. >> *(1) For FD discretization* >> >> *A |---dx--| B* >> *|-------|-------|-------|-------|-------| grid size = nx* >> *0 i-1 i i+1 nx-1* >> >> the vertex-centered FD discretization of the domain is >> >> *x(i) = i*dx (i = 0,..., nx)), where dx = 1/nx* >> >> and the RHS function G of the equation is >> >> *G[i] = (u[i+1]-2*u[i]+u[i-1])/dx^2* >> >> There have been dozens of examples demonstrating way of BC setting with a >> finite difference medthod, e.g., >> >> https://github.com/petsc/petsc/blob/386ebab930a846575609b49074a40c46e7f8ed75/src/ts/tutorials/ex26.c#L293-L343 >> which I think I have understood how it works and the following lines of >> code evaluate the RHS function *G(t,u)* in PETSc's *TSSetRHSFunction()* >> routine >> /*------------------FD discretization-------------------------*/ >> *for (i = xs; i < xs + xm; i++) {* >> >> *// left boundary* >> *if (i == 0) {* >> >> *G[i] = u[i] - u_A; **// Dirichlet BC: u(x=0) = u_A;* >> *G[i] = (u[i] - u[i+1])/dx - q_A; **// Neumann BC: du/dx(x=0.0) = >> q_A;* >> >> *} * >> *// right boundary* >> *else if (i == mx - 1) {* >> >> *G[i] = u[i] - u_B; **// Dirichlet BC: u(x=1.0) = u_B;* >> *G[i] = (u[i]-u[i-1])/dx - q_B; **// Neumann BC: du/dx(x=0.0) = >> q_A;* >> >> *}* >> *// internal* >> *else {* >> >> *G[i] = (u[i-1] - 2 * u[i] + u[i + 1])/dx^2; **//u_xx* >> >> *}* >> >> *}**/*------------------------------------------------------------*/* >> >> The BCs, e.g., at the left boudnary, can be straightforwardly set as >> follows: >> Dirichlet BC: >> *G[0] = u[0] - u_A* >> which indicates *G[0] = u[0] - u_A = 0* and Dirichlet BC *u[0] = u_A* is >> thus applied >> Neumann BC: >> *G[0] = (u[0] - u[1])/dx - q_A* >> which indicates *G[0] = (u[0] - u[1])/dx - q_A = 0* and Neumann BC *du/dx >> = (u[0] - u[1])/dx = q_A* is applied >> This is the way the above examples apply BCs. >> >> For a structured, cell-centered FVM discretization, however, I tried and >> found the BCs can not be applied in a similar way. I will explain how I >> tried this as follows. >> *(2) FV discretization* >> >> *A |---dx--| B* >> *|---o---|---o---|---o---|---o---|---o---| cell size = nx* >> *W w P e E * >> *0 i-1 i i+1 nx-1* >> >> (NOTE: the center cell is denoted as P and its neighbouring cells as W, >> E; cell faces are denoted as w, e; the left and right boundary faces are >> A and B, respectively.) >> >> The cell-centered FV discretization of the domain is >> >> *x(i) = dx/2 + i*dx (i = 0,..., nx-1)), where dx = 1/nx* >> >> the diffusion term u_xx can be reorganized by using divergence theorem >> >> *(u_E-u_P) (u_P-u_W)* >> *--------- - ---------* >> * dx dx* >> >> and the RHS function G of the equation is >> >> *G[i] = (u_E-u_P)/dx - (u_P-u_W)/dx = (u[i+1]-2*u[i]+u[i-1])/dx* >> >> the key lines of code in RHS function G(t,u) in PETSc's >> *TSSetRHSFunction()* routine can be >> >> */*------------------FV discretization-------------------------*/* >> *for (i = xs; i < xs + xm; i++) {* >> >> *// left boundary* >> *if (i == 0) {* >> >> *G[i] = 3.0 * u[i] - u[i+1] - 2.0 * u_A; **//Dirichlet BC: u(x=0.0) = >> u_A, does not work* >> *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) >> = q_A;* >> >> *}* >> *// right boundary* >> *else if (i == mx - 1) {* >> >> *G[i] = 3.0 * u[i] - u[i-1] - 2.0 * u_B; **//Dirichlet BC: u(x=1.0) = >> u_B, does not work* >> *G[i] = (u[i]-u[i-1])/dx - q_B; **//Neumann BC: du/dx(x=1.0) >> = q_B, does not work* >> >> *}* >> *// internal* >> *else {* >> >> *G[i] = (u[i-1] - 2*u[i] + u[i+1])/dx; **//u_x* >> >> *}* >> >> *}**/*------------------------------------------------------------*/* >> >> I tried to apply BCs in a way similar to that of the FD discretization. >> For the Dirichlet BC on the left boundary, because of the cell-centered >> arrangement of u[i], the left BC is given as *u_A* at cell face A >> instead of cell center. >> So, I first applied a linear interpolation to get the cell-centered value >> of *u[0]* from the left boundary value *u_A* and neighbouring *u[1]* >> >> *u[0] = (u[1]+2*u_A)/3.0 (equivalent to 3*u[0] - u[1] - 2*u_A = 0.0)* >> >> * |-dx/2-|------dx-----|* >> * |------o------|------o------| linear interpolation: u[0] = >> (u[1]+2*u_A)/3.0* >> * A P e E* >> * u_A u[0] u[1]* >> >> Dirichlet BC: >> *G[0] = 3.0*u[0] - u[1] - 2.0*u_A* >> in which *u[0] = (u[1]+2*u_A)/3.0* is implicitly calculated and I >> expect the Dirichlet BC *u(x=0) = u_A* is applied >> >> For the Neumann BC, left BC is given as a gradient of *u*, *q_A*, at >> cell face A >> Neumann BC: >> *G[i] = (u[1] - u[0])/dx - q_A* >> in which Neumann BC *du/dx = (u[1] - u[0])/dx = q_A* is expected to >> be applied. >> >> However, this FV discretization failed to work. Is there anything wrong >> or could you give me a hint? >> >> Best, >> Qw >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From zhfreewill at gmail.com Mon Nov 29 06:28:11 2021 From: zhfreewill at gmail.com (zhfreewill) Date: Mon, 29 Nov 2021 20:28:11 +0800 Subject: [petsc-users] How to set boundary conditions for a structured finite volume (FV) discretization when using TS/SNES? In-Reply-To: References: <8ABB6088-B147-40BC-BC6D-62438B28FB33@petsc.dev> Message-ID: Thank you all! I think I did it. I used the ghost cells and the code behaves properly. Attached is my document with detailed derivation and the code. Best, Qw Mark Adams ?2021?11?28??? ??10:17??? > * I have used Matt's approach and it works but you do have to remember to > update the ghost cells after you update your solution and before you > use the operator again. > * I would use a convergence test to determine if the result is correct. > * Check that a constant function is a (trivial) solution to your > discretization. > > On Sat, Nov 27, 2021 at 2:54 AM zhfreewill wrote: > >> Hi,Barry, >> Thanks for reply. I meant I cannot get the right result as the FD >> version did, although the code did not report any error when running. >> >> Indeed, there were mistakes and I fixed it in the code (I also attached >> additional derivation and code this time, please see the attachment). >> Still, I cannot get the right output so far. >> >> Regards, >> Qw >> >> Barry Smith ?2021?11?26??? ??11:31??? >> >>> >>> Could you be more specific in saying how it fails (can you share the >>> actual prototype code?) >>> >>> Note here you seem to be missing a /dx term >>> >>> *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) >>> = q_A;* >>> >>> >>> Barry >>> >>> On Nov 26, 2021, at 4:02 AM, zhfreewill wrote: >>> >>> Hi all, >>> I have a basic question on how to apply B.C. for a cell-centered >>> structured FV discretization when using the SNES and Ts object. >>> For instance, for a 1-D transient diffusion equation >>> >>> *u_t = u_xx, 0 < x < 1* >>> >>> with IC: >>> >>> *u(x,t=0) = 0;* >>> >>> and Dirichlet BC: >>> >>> *u(x=0) = u_A = 0.0;* >>> *u(x=1) = u_B = 1.0;* >>> >>> or Neumann BC: >>> >>> *du/dx(x=0.0) = q_A = 0.0;* >>> *du/dx(x=1.0) = q_B = 1.0;* >>> >>> >>> When using TS of PETSc, the equation can be organized in form of >>> *U_t = G(t,u)* >>> where *G* is the right-hand-side (RHS) function regardless of the >>> specific method of discretization. >>> *(1) For FD discretization* >>> >>> *A |---dx--| B* >>> *|-------|-------|-------|-------|-------| grid size = nx* >>> *0 i-1 i i+1 nx-1* >>> >>> the vertex-centered FD discretization of the domain is >>> >>> *x(i) = i*dx (i = 0,..., nx)), where dx = 1/nx* >>> >>> and the RHS function G of the equation is >>> >>> *G[i] = (u[i+1]-2*u[i]+u[i-1])/dx^2* >>> >>> There have been dozens of examples demonstrating way of BC setting with >>> a finite difference medthod, e.g., >>> >>> https://github.com/petsc/petsc/blob/386ebab930a846575609b49074a40c46e7f8ed75/src/ts/tutorials/ex26.c#L293-L343 >>> which I think I have understood how it works and the following lines of >>> code evaluate the RHS function *G(t,u)* in PETSc's *TSSetRHSFunction()* >>> routine >>> /*------------------FD discretization-------------------------*/ >>> *for (i = xs; i < xs + xm; i++) {* >>> >>> *// left boundary* >>> *if (i == 0) {* >>> >>> *G[i] = u[i] - u_A; **// Dirichlet BC: u(x=0) = u_A;* >>> *G[i] = (u[i] - u[i+1])/dx - q_A; **// Neumann BC: du/dx(x=0.0) = >>> q_A;* >>> >>> *} * >>> *// right boundary* >>> *else if (i == mx - 1) {* >>> >>> *G[i] = u[i] - u_B; **// Dirichlet BC: u(x=1.0) = u_B;* >>> *G[i] = (u[i]-u[i-1])/dx - q_B; **// Neumann BC: du/dx(x=0.0) = >>> q_A;* >>> >>> *}* >>> *// internal* >>> *else {* >>> >>> *G[i] = (u[i-1] - 2 * u[i] + u[i + 1])/dx^2; **//u_xx* >>> >>> *}* >>> >>> *}**/*------------------------------------------------------------*/* >>> >>> The BCs, e.g., at the left boudnary, can be straightforwardly set as >>> follows: >>> Dirichlet BC: >>> *G[0] = u[0] - u_A* >>> which indicates *G[0] = u[0] - u_A = 0* and Dirichlet BC *u[0] = >>> u_A* is thus applied >>> Neumann BC: >>> *G[0] = (u[0] - u[1])/dx - q_A* >>> which indicates *G[0] = (u[0] - u[1])/dx - q_A = 0* and Neumann BC *du/dx >>> = (u[0] - u[1])/dx = q_A* is applied >>> This is the way the above examples apply BCs. >>> >>> For a structured, cell-centered FVM discretization, however, I tried and >>> found the BCs can not be applied in a similar way. I will explain how I >>> tried this as follows. >>> *(2) FV discretization* >>> >>> *A |---dx--| B* >>> *|---o---|---o---|---o---|---o---|---o---| cell size = nx* >>> *W w P e E * >>> *0 i-1 i i+1 nx-1* >>> >>> (NOTE: the center cell is denoted as P and its neighbouring cells as W, >>> E; cell faces are denoted as w, e; the left and right boundary faces >>> are A and B, respectively.) >>> >>> The cell-centered FV discretization of the domain is >>> >>> *x(i) = dx/2 + i*dx (i = 0,..., nx-1)), where dx = 1/nx* >>> >>> the diffusion term u_xx can be reorganized by using divergence theorem >>> >>> *(u_E-u_P) (u_P-u_W)* >>> *--------- - ---------* >>> * dx dx* >>> >>> and the RHS function G of the equation is >>> >>> *G[i] = (u_E-u_P)/dx - (u_P-u_W)/dx = (u[i+1]-2*u[i]+u[i-1])/dx* >>> >>> the key lines of code in RHS function G(t,u) in PETSc's >>> *TSSetRHSFunction()* routine can be >>> >>> */*------------------FV discretization-------------------------*/* >>> *for (i = xs; i < xs + xm; i++) {* >>> >>> *// left boundary* >>> *if (i == 0) {* >>> >>> *G[i] = 3.0 * u[i] - u[i+1] - 2.0 * u_A; **//Dirichlet BC: u(x=0.0) = >>> u_A, does not work* >>> *G[i] = (u[i+1] - u[i]) - q_A; **//Neumann BC: du/dx(x=0.0) >>> = q_A;* >>> >>> *}* >>> *// right boundary* >>> *else if (i == mx - 1) {* >>> >>> *G[i] = 3.0 * u[i] - u[i-1] - 2.0 * u_B; **//Dirichlet BC: u(x=1.0) = >>> u_B, does not work* >>> *G[i] = (u[i]-u[i-1])/dx - q_B; **//Neumann BC: du/dx(x=1.0) >>> = q_B, does not work* >>> >>> *}* >>> *// internal* >>> *else {* >>> >>> *G[i] = (u[i-1] - 2*u[i] + u[i+1])/dx; **//u_x* >>> >>> *}* >>> >>> *}**/*------------------------------------------------------------*/* >>> >>> I tried to apply BCs in a way similar to that of the FD discretization. >>> For the Dirichlet BC on the left boundary, because of the cell-centered >>> arrangement of u[i], the left BC is given as *u_A* at cell face A >>> instead of cell center. >>> So, I first applied a linear interpolation to get the cell-centered >>> value of *u[0]* from the left boundary value *u_A* and neighbouring >>> *u[1]* >>> >>> *u[0] = (u[1]+2*u_A)/3.0 (equivalent to 3*u[0] - u[1] - 2*u_A = >>> 0.0)* >>> >>> * |-dx/2-|------dx-----|* >>> * |------o------|------o------| linear interpolation: u[0] = >>> (u[1]+2*u_A)/3.0* >>> * A P e E* >>> * u_A u[0] u[1]* >>> >>> Dirichlet BC: >>> *G[0] = 3.0*u[0] - u[1] - 2.0*u_A* >>> in which *u[0] = (u[1]+2*u_A)/3.0* is implicitly calculated and I >>> expect the Dirichlet BC *u(x=0) = u_A* is applied >>> >>> For the Neumann BC, left BC is given as a gradient of *u*, *q_A*, at >>> cell face A >>> Neumann BC: >>> *G[i] = (u[1] - u[0])/dx - q_A* >>> in which Neumann BC *du/dx = (u[1] - u[0])/dx = q_A* is expected to >>> be applied. >>> >>> However, this FV discretization failed to work. Is there anything wrong >>> or could you give me a hint? >>> >>> Best, >>> Qw >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: UnsteadyDiffusion1D_ver2.c Type: application/octet-stream Size: 6424 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Diffusion_FV_derivation.pdf Type: application/pdf Size: 2962554 bytes Desc: not available URL: From knepley at gmail.com Mon Nov 29 09:02:05 2021 From: knepley at gmail.com (Matthew Knepley) Date: Mon, 29 Nov 2021 10:02:05 -0500 Subject: [petsc-users] PETSc Ordering for vectors generated by DMCreateGlobalVector In-Reply-To: References: Message-ID: On Sun, Nov 28, 2021 at 8:05 AM ?? wrote: > Hi all, > > Application Ordering of a vector is generally different from its PETSc > one. In my application, I wish to retrieve the value of a vector generated > by DMCreateGlobalVector into its original application order. I know there > are some functions like DMDAGetAO to do this. My question is : Is there any > DMPlex version of such a function? If not, I wish to know how to do it. > There is a GlobalToNatural map for Plex. We are in the process of reworking this, but you can see its use in Plex test ex26. Thanks, Matt > Many thanks > > Yuan > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mfadams at lbl.gov Mon Nov 29 19:03:40 2021 From: mfadams at lbl.gov (Mark Adams) Date: Mon, 29 Nov 2021 20:03:40 -0500 Subject: [petsc-users] query device matrix for what it is up-to-date Message-ID: If I ask for a kokkos matrix norm, is there a way to know if it is using the device or host data? I assume they are both up to date after the norm unless it calls a GPU norm method, but I would like to know which memory was "active" before the norm. And does anyone know if aijcusparse matrices have the same semantics? Thanks, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrsd at gmail.com Tue Nov 30 08:50:24 2021 From: andrsd at gmail.com (David Andrs) Date: Tue, 30 Nov 2021 07:50:24 -0700 Subject: [petsc-users] Output data using ExodusIIViewer Message-ID: Hello! I am trying to store data into an ExodusII file using the ExodusIIViewer, but running into a segfault inside PETSc. Attached is a minimal example showing the problem. It can very much be that I am missing something obvious. However, if I change the code to VTKViewer I get the desired output file. Machine: MacBook Pro 2019 OS version/type: Darwin notak.local 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64 PETSc: Petsc Release Version 3.16.1, Nov 01, 2021 MPI: MPICH 3.4.2 Compiler: clang-12 Call stack (not sure how relevant that is since it is from opt version): * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xc) frame #0: 0x0000000102303ba9 libpetsc.3.16.dylib`DMView_PlexExodusII(dm=, viewer=) at plexexodusii.c:457:45 [opt] 454 else if (degree == 2) nodes[cs] = nodesHexP2; 455 } 456 /* Compute the number of cells not in the connectivity table */ -> 457 cellsNotInConnectivity -= nodes[cs][3]*csSize; 458 459 ierr = ISRestoreIndices(stratumIS, &cells);CHKERRQ(ierr); 460 ierr = ISDestroy(&stratumIS);CHKERRQ(ierr); With regards, David Andrs -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: exo2.c Type: application/octet-stream Size: 1219 bytes Desc: not available URL: From bourdin at mcmaster.ca Tue Nov 30 12:39:02 2021 From: bourdin at mcmaster.ca (Blaise Bourdin) Date: Tue, 30 Nov 2021 18:39:02 +0000 Subject: [petsc-users] Output data using ExodusIIViewer In-Reply-To: References: Message-ID: <48B7C2BC-B133-4ADB-A269-56B666A52C81@mcmaster.ca> An HTML attachment was scrubbed... URL: From yuanxi at advancesoft.jp Tue Nov 30 18:18:35 2021 From: yuanxi at advancesoft.jp (=?UTF-8?B?6KKB54WV?=) Date: Wed, 1 Dec 2021 09:18:35 +0900 Subject: [petsc-users] PETSc Ordering for vectors generated by DMCreateGlobalVector In-Reply-To: References: Message-ID: I see. It would be a feasible approach to solve my problem. Thank you for your suggestion. Yuan 2021?11?30?(?) 0:02 Matthew Knepley : > On Sun, Nov 28, 2021 at 8:05 AM ?? wrote: > >> Hi all, >> >> Application Ordering of a vector is generally different from its PETSc >> one. In my application, I wish to retrieve the value of a vector generated >> by DMCreateGlobalVector into its original application order. I know there >> are some functions like DMDAGetAO to do this. My question is : Is there any >> DMPlex version of such a function? If not, I wish to know how to do it. >> > > There is a GlobalToNatural map for Plex. We are in the process of > reworking this, but you can see its use in Plex test ex26. > > Thanks, > > Matt > > >> Many thanks >> >> Yuan >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrsd at gmail.com Tue Nov 30 18:31:33 2021 From: andrsd at gmail.com (David Andrs) Date: Tue, 30 Nov 2021 17:31:33 -0700 Subject: [petsc-users] Output data using ExodusIIViewer In-Reply-To: <48B7C2BC-B133-4ADB-A269-56B666A52C81@mcmaster.ca> References: <48B7C2BC-B133-4ADB-A269-56B666A52C81@mcmaster.ca> Message-ID: <7d14bab6-7df9-4804-9bb5-c810c64a7a86@edison.tech> I see. I added a "Cell Sets" label like so: ? ? DMCreateLabel(dm, "Cell Sets"); DMLabel cs_label; DMGetLabel(dm, "Cell Sets", &cs_label); DMLabelAddStratum(cs_label, 0); PetscInt idxs[] = { 1 }; IS is; ISCreateGeneral(comm, 1, idxs, PETSC_COPY_VALUES, &is); DMLabelSetStratumIS(cs_label, 0, is); ISDestroy(&is); Note, that I have only a single element (Quad4) in the mesh and I am just trying to get this working, so I understand what needs to happen for larger meshes. ? ?This got me past the segfault, but now I see: ? [?0]PETSC ERROR: Argument out of range [0]PETSC ERROR: Number of vertices 1 in dimension 2 has no ExodusII type So, I assume I need to do something ?more to make it work. ? ?David ?? On Nov 30 2021, at 11:39 AM, Blaise Bourdin wrote: > > It looks like your DM cannot be saved in exodus format as such. The exodus format requires that all cells be part of a single block (defined by ?Cell Set? labels), and that the cell sets consists of sequentially numbered cells. > > Can you see if that is enough? If not, I will go through your example > > > > Blaise > > > > > > > > On Nov 30, 2021, at 9:50 AM, David Andrs wrote: > > > > > > > > > > Hello! > > > > > > > > I am trying to store data into an ExodusII file using the ExodusIIViewer, but running into a segfault inside PETSc. Attached is a minimal example showing the problem. It can very much be that I am missing something obvious. However, if I change the code to VTKViewer I get the desired output file. > > > > > > > > Machine: MacBook Pro 2019 > > > > OS version/type: Darwin notak.local 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64 x86_64 > > > > PETSc: Petsc Release Version 3.16.1, Nov 01, 2021 > > > > MPI: MPICH 3.4.2 > > > > Compiler: clang-12 > > > > > > > > Call stack (not sure how relevant that is since it is from opt version): > > > > > > > > * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xc) > > > > frame #0: 0x0000000102303ba9 libpetsc.3.16.dylib`DMView_PlexExodusII(dm=, viewer=) at plexexodusii.c:457:45 [opt] > > > > 454 else if (degree == 2) nodes[cs] = nodesHexP2; > > > > 455 } > > > > 456 /* Compute the number of cells not in the connectivity table */ > > > > -> 457 cellsNotInConnectivity -= nodes[cs][3]*csSize; > > > > 458 > > > > 459 ierr = ISRestoreIndices(stratumIS, &cells);CHKERRQ(ierr); > > > > 460 ierr = ISDestroy(&stratumIS);CHKERRQ(ierr); > > > > > > > > > > > > With regards, > > > > > > > > > > David Andrs > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Professor, Department of Mathematics & Statistics > > Hamilton Hall room 409A, McMaster University > > 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada > > Tel. +1 (905) 525 9140 ext. 27243 > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: