[petsc-users] Compiling with PETSc 64-bit indices

[TAY wee-beng]

On 21/2/2018 9:00 AM, Matthew Knepley wrote:
> On Tue, Feb 20, 2018 at 7:54 PM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     When I run my CFD code with a grid size of 1119x1119x499 ( total
>     grid size =    624828339 ), I got the error saying I need to
>     compile PETSc with 64-bit indices.
>
>     So I tried to compile PETSc again and then compile my CFD code
>     with the newly compiled PETSc. However, now I got segmentation error:
>
>     rm: cannot remove `log': No such file or directory
>     [409]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [409]PETSC ERROR: [535]PETSC ERROR: [410]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [410]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     [410]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     [410]PETSC ERROR: [536]PETSC ERROR:
>     ------------------------------------------------------------------------
>     [536]PETSC ERROR: Caught signal number 11 SEGV: Segmentation
>     Violation, probably memory access out of range
>     [536]PETSC ERROR: Try option -start_in_debugger or
>     -on_error_attach_debugger
>     [536]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>     <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>     [536]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>     Apple Mac OS X to find memory corruption errors
>     [536]PETSC ERROR: likely location of problem given in stack below
>     [536]PETSC ERROR: ---------------------  Stack Frames
>     ------------------------------------
>     [536]PETSC ERROR: Note: The EXACT line numbers in the stack are
>     not available,
>     [536]PETSC ERROR:       INSTEAD the line number of the start of
>     the function
>     [536]PETSC ERROR:       is given.
>     [536]PETSC ERROR: [536] DMDACheckOwnershipRanges_Private line 581
>     /home/users/nus/tsltaywb/source/petsc-3.7.6/src/dm/impls/da/da.c
>     [536]PETSC ERROR: or see
>     http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>     <http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind>
>     [410]PETSC ERROR: or try http://valgrind.org on GNU/linux and
>     Apple Mac OS X to find memory corruption errors
>     [410]PETSC ERROR: likely location of problem given in stack below
>     [410]PETSC ERROR: ---------------------  Stack Frames
>     ------------------------------------
>     [410]PETSC ERROR: Note: The EXACT line numbers in the stack are
>     not available,
>     [897]PETSC ERROR: [536] DMDASetOwnershipRanges line 613
>     /home/users/nus/tsltaywb/source/petsc-3.7.6/src/dm/impls/da/da.c
>     [536]PETSC ERROR: [536] DMDACreate3d line 1434
>     /home/users/nus/tsltaywb/source/petsc-3.7.6/src/dm/impls/da/da3.c
>     [536]PETSC ERROR: --------------------- Error Message
>     --------------------------------------------------------------
>
>     The CFD code worked previously but increasing the problem size
>     results in segmentation error. It seems to be related to
>     DMDACreate3d and DMDASetOwnershipRanges. Any idea where the
>     problem lies?
>
>     Besides, I want to know when and why do I have to use PETSc with
>     64-bit indices?
>
>
> 1) A 32-bit integer can hold numbers up to 2^32 = 4.2e9, so if you 
> have a 3D velocity, pressure, and energy, you already have 3e9 unknowns,
>     before you even start to count nonzero entries in the matrix. 
> 64-bit integers allow you to handle these big sizes.
>
>     Also, can I use the 64-bit indices version with smaller sized
>     problems?
>
>
> 2) Yes
>
>     And is there a speed difference between using the 32-bit and
>     64-bit indices ver?
>
>
> 3) I have seen no evidence of this
>
> 4) My guess is that you have defines regular integers in your code and 
> passed them to PETSc, rather than using PetscInt as the type.
Oh that seems probable. So I am still using integer(4) when it should be 
integer(8) for some values, is that so? If I use PetscInt, is it the 
same as integer(8)? Or does it depend on the actual number?

I wonder if I replace all my integer to PetscInt, will there be a large 
increase in memory usage, because all integer(4) now becomes integer(8)?

Thanks.
>
>   Thanks,
>
>      Matt
>
>
>     -- 
>     Thank you very much.
>
>     Yours sincerely,
>
>     ================================================
>     TAY Wee-Beng (Zheng Weiming) 郑伟明
>     Personal research webpage: http://tayweebeng.wixsite.com/website
>     <http://tayweebeng.wixsite.com/website>
>     Youtube research showcase:
>     https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA
>     <https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA>
>     linkedin: www.linkedin.com/in/tay-weebeng
>     <http://www.linkedin.com/in/tay-weebeng>
>     ================================================
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180221/e75264dd/attachment.html>