<div dir="ltr"><div dir="ltr">On Mon, Feb 25, 2019 at 3:19 AM Appel, Thibaut <<a href="mailto:t.appel17@imperial.ac.uk">t.appel17@imperial.ac.uk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
Hi Matthew,
<div><br>
</div>
<div>Yes I need F90 and the syntax in the file / yours</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<blockquote type="cite"><font color="#000000"><span style="background-color:rgba(255,255,255,0)">PetscInt, pointer :: id_ltog(:)</span></font></blockquote>
</div>
</div>
</div>
<div><br>
</div>
Is the exact same as</div>
<div>
<blockquote type="cite">PetscInt, dimension(:), pointer :: id_ltog</blockquote>
</div>
<div><br>
</div>
<div>They’re both modern fortran ”correct”</div>
<div><br>
</div>
<div>Anyways I tried both and I still get the same error message.</div></div></blockquote><div><br></div><div>Okay, that file I sent the link to is tested every night, so I suspect a compiler bug in your version.</div><div>Please send your entire example so I can run it here. Also, let us know what Fortran compiler</div><div>and version you are using again.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">
<div>Thibaut<br>
<div dir="ltr"><br>
On 24 Feb 2019, at 23:38, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On Sun, Feb 24, 2019 at 6:33 PM Appel, Thibaut <<a href="mailto:t.appel17@imperial.ac.uk" target="_blank">t.appel17@imperial.ac.uk</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">Is that for the id_ltog argument? I tried declaring it as IS, and you can’t declare an deferred-shape array as target. Same message. XXX(:) is syntactic sugar for dimension(:) :: XXX
<div><br>
</div>
<div>I’m really confused because this</div>
<div><a href="https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex14f.F90.html</a></div>
<div>doesn’t use the xxxF90 variants of the routines we’re talking about?</div>
</div>
</blockquote>
<div><br>
</div>
<div>You can use the F77 stuff if you want. I thought you needed F90. For F90, I think you need to declare it</div>
<div><br>
</div>
<div> PetscInt, pointer :: id_ltog(:)</div>
<div><br>
</div>
<div>like it is in the link I sent you.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">
<div>Thibaut<br>
<div><br>
<blockquote type="cite">
<div>On 24 Feb 2019, at 22:50, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br class="gmail-m_5832061864260648120gmail-m_3669154753077176379Apple-interchange-newline">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">On Sun, Feb 24, 2019 at 4:28 PM Appel, Thibaut <<a href="mailto:t.appel17@imperial.ac.uk" target="_blank">t.appel17@imperial.ac.uk</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word"><span>Hi Matthew,</span><span><br>
</span><span><br>
</span><span>With the following data declaration and routines calls:</span></div>
</blockquote>
<div><br>
</div>
<div>I am not a Fortran90 expert, but your declaration looks different than the ones I have seen work:</div>
<div><br>
</div>
<div> <a href="https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90" target="_blank">https://bitbucket.org/petsc/petsc/src/master/src/dm/impls/plex/examples/tutorials/ex1f90.F90</a></div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word"><font face="Courier"><span><br>
</span><span> DM :: da<br>
</span><span> ISLocalToGlobalMapping :: ltog<br>
</span><span> PetscInt, DIMENSION(:), POINTER :: id_ltog<br>
<br>
CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &<br>
PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL DMSetUp(da,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL DMGetLocalToGlobalMapping(da,ltog,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL ISLocalToGlobalMappingGetIndicesF90(ltog,id_ltog,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL ISLocalToGlobalMappingRestoreIndicesF90(ltog,id_ltog,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL DMDestroy(da,ierr)<br>
CHKERRA(ierr)<br>
</span></font><br>
<br>
I get, with the most recent Intel Fortran compiler and a fresh “git pull” PETsc:<span><br>
<br>
</span><span><font face="Courier">forrtl: severe (408): fort: (7): Attempt to use pointer ID_LTOG when it is not associated with a target</font></span>
<div><span><font face="Courier"><br>
</font></span></div>
<div><span><font face="Courier"><br>
</font></span></div>
<div><span>Thibaut<br>
<br>
<blockquote type="cite">On 22 Feb 2019, at 15:13, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
<br>
On Fri, Feb 22, 2019 at 9:10 AM Thibaut Appel via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
I reckon that what corresponds best to my situation is to create my own MPIAIJ matrix, recover the DMDA local to global mapping, preallocate myself and fill my values with MatSetValues.<br>
<br>
However, what is the correct way to call ISLocalToGlobalMappingGetIndices in Fortran? It seems there's something missing in the documentation as my code just won't work.<br>
<br>
Some examples include a PetscOffset argument which is not present in the documentation and there is no manual page for ISLocalToGlobalMappingGetIndicesF90: the only information I could find is on /src/vec/is/utils/f90-custom/zisltogf90.c. Are you supposed to
allocate the index array yourself? I tried declaring it as a pointer too. I keep getting an error<br>
<br>
1) There was a bug in that file, but I don't think it should have affected you. I will put in a PR<br>
<br>
2) You are supposed to pass in a pointer, unallocated. We manage all the allocation, which is why you call Restore. What happens when you do that?<br>
<br>
Thanks,<br>
<br>
Matt<br>
[0]PETSC ERROR: #1 User provided function() line 0 in User file<br>
application called MPI_Abort(MPI_COMM_SELF, -42924) - process 0<br>
<br>
Minimal example:<br>
<br>
PROGRAM test_dmda<br>
<br>
#include <petsc/finclude/petscdm.h><br>
#include <petsc/finclude/petscdmda.h><br>
<br>
USE PetscDMDA<br>
<br>
IMPLICIT NONE<br>
<br>
PetscErrorCode :: ierr<br>
PetscInt :: irank, icx, icy, lx, ly, n_ltog<br>
DM :: da<br>
ISLocalToGlobalMapping :: ltog<br>
<br>
PetscInt, DIMENSION(:), ALLOCATABLE :: id_ltog<br>
!PetscInt, DIMENSION(:), POINTER :: id_ltog<br>
<br>
PetscInt, PARAMETER :: nx=24, ny=9, neq=4<br>
CHARACTER(LEN=100) :: err_msg<br>
<br>
CALL PetscInitialize(PETSC_NULL_CHARACTER,ierr)<br>
<br>
CALL MPI_COMM_RANK(PETSC_COMM_WORLD,irank,ierr)<br>
<br>
CALL DMDACreate2D(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,(nx+1),(ny+1), &<br>
PETSC_DECIDE,PETSC_DECIDE,neq,0,PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL DMSetUp(da,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL DMDAGetCorners(da,icx,icy,PETSC_NULL_INTEGER,lx,ly,PETSC_NULL_INTEGER,ierr)<br>
CHKERRA(ierr)<br>
<br>
WRITE(*,'(*(1X,A,I3))') 'I am process #', irank, ' - my lower left corner is icx=', icx, &<br>
'and icy=', icy, ' - the width is lx=', lx, ' and ly=', ly<br>
<br>
CALL DMGetLocalToGlobalMapping(da,ltog,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL ISLocalToGlobalMappingGetSize(ltog,n_ltog,ierr)<br>
CHKERRA(ierr)<br>
<br>
ALLOCATE(id_ltog(1:n_ltog),STAT=ierr,ERRMSG=err_msg)<br>
<br>
CALL ISLocalToGlobalMappingGetIndices(ltog,id_ltog,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL ISLocalToGlobalMappingRestoreIndices(ltog,id_ltog,ierr)<br>
CHKERRA(ierr)<br>
<br>
DEALLOCATE(id_ltog,STAT=ierr,ERRMSG=err_msg)<br>
<br>
CALL DMDestroy(da,ierr)<br>
CHKERRA(ierr)<br>
<br>
CALL PetscFinalize(ierr)<br>
<br>
END PROGRAM test_dmda<br>
<br>
<br>
<br>
On 21/02/2019 17:49, Matthew Knepley wrote:<br>
<blockquote type="cite">On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
Dear PETSc developers/users,<br>
<br>
I’m solving linear PDEs on a regular grid with high-order finite differences, assembling an MPIAIJ matrix to solve linear systems or eigenvalue problems. I’ve been using vertex major, natural ordering for the parallelism with PetscSplitOwnership (yielding rectangular
slices of the physical domain) and wanted to move to DMDA to have a more square-ish domain decomposition and minimize communication between processes.<br>
<br>
However, my application is memory critical, and I have finely-tuned matrix preallocation routines for allocating memory “optimally”. It seems the memory of a DMDA matrix is allocated along the value of the stencil width of DMDACreate and the manual says about
it<br>
<br>
“These DMDA stencils have nothing directly to do with any finite difference stencils one might chose to use for a discretization”<br>
<br>
And despite reading the manual pages there must be something I do not understand in the DM topology, what is that "stencil width" for then? I will not use ghost values for my FD-method, right?<br>
<br>
What this is saying is, "You might be using some stencil that is not STAR or BOX, but we are preallocating according to one of those".<br>
If you really care about how much memory is preallocated, which it seems you do, then you might be able to use<br>
<br>
<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html</a><br>
<br>
to tell use exactly how to preallocate.<br>
<br>
I was then wondering if I could just create a MPIAIJ matrix, and with a PETSc routine get the global indices of the domain for each process: in other words, an equivalent of PetscSplitOwnership that gives me the DMDA unknown ordering. So I can feed and loop
on that in my preallocation and assembly routines.<br>
<br>
You can make an MPIAIJ matrix yourself of course. It should have the same division of rows as the DMDA division of dofs. Also, MatSetValuesStencil() will not work for a custom matrix.<br>
<br>
Thanks,<br>
<br>
Matt<br>
<br>
Thanks very much,<br>
<br>
Thibaut<br>
<br>
<br>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
<br>
<a href="https://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</blockquote>
<br>
<br>
-- <br>
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>
<br>
<a href="https://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</blockquote>
<br>
</span></div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_5832061864260648120gmail-m_3669154753077176379gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_5832061864260648120gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>