[petsc-users] Compiling with PETSc 64-bit indices

[Matthew Knepley]

On Tue, Feb 20, 2018 at 8:08 PM, TAY wee-beng <zonexo at gmail.com> wrote:

>
> On 21/2/2018 9:00 AM, Matthew Knepley wrote:
>
> On Tue, Feb 20, 2018 at 7:54 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>> Hi,
>>
>> When I run my CFD code with a grid size of 1119x1119x499 ( total grid
>> size =    624828339 ), I got the error saying I need to compile PETSc with
>> 64-bit indices.
>>
>> So I tried to compile PETSc again and then compile my CFD code with the
>> newly compiled PETSc. However, now I got segmentation error:
>>
>> rm: cannot remove `log': No such file or directory
>> [409]PETSC ERROR: ------------------------------
>> ------------------------------------------
>> [409]PETSC ERROR: [535]PETSC ERROR: [410]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [410]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [410]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> [410]PETSC ERROR: [536]PETSC ERROR: ------------------------------
>> ------------------------------------------
>> [536]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> [536]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> [536]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d
>> ocumentation/faq.html#valgrind
>> [536]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>> OS X to find memory corruption errors
>> [536]PETSC ERROR: likely location of problem given in stack below
>> [536]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [536]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [536]PETSC ERROR:       INSTEAD the line number of the start of the
>> function
>> [536]PETSC ERROR:       is given.
>> [536]PETSC ERROR: [536] DMDACheckOwnershipRanges_Private line 581
>> /home/users/nus/tsltaywb/source/petsc-3.7.6/src/dm/impls/da/da.c
>> [536]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d
>> ocumentation/faq.html#valgrind
>> [410]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>> OS X to find memory corruption errors
>> [410]PETSC ERROR: likely location of problem given in stack below
>> [410]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> [410]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> [897]PETSC ERROR: [536] DMDASetOwnershipRanges line 613
>> /home/users/nus/tsltaywb/source/petsc-3.7.6/src/dm/impls/da/da.c
>> [536]PETSC ERROR: [536] DMDACreate3d line 1434
>> /home/users/nus/tsltaywb/source/petsc-3.7.6/src/dm/impls/da/da3.c
>> [536]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>>
>> The CFD code worked previously but increasing the problem size results in
>> segmentation error. It seems to be related to DMDACreate3d and
>> DMDASetOwnershipRanges. Any idea where the problem lies?
>>
>> Besides, I want to know when and why do I have to use PETSc with 64-bit
>> indices?
>>
>
> 1) A 32-bit integer can hold numbers up to 2^32 = 4.2e9, so if you have a
> 3D velocity, pressure, and energy, you already have 3e9 unknowns,
>     before you even start to count nonzero entries in the matrix. 64-bit
> integers allow you to handle these big sizes.
>
>
>> Also, can I use the 64-bit indices version with smaller sized problems?
>>
>
> 2) Yes
>
>
>> And is there a speed difference between using the 32-bit and 64-bit
>> indices ver?
>
>
> 3) I have seen no evidence of this
>
> 4) My guess is that you have defines regular integers in your code and
> passed them to PETSc, rather than using PetscInt as the type.
>
> Oh that seems probable. So I am still using integer(4) when it should be
> integer(8) for some values, is that so? If I use PetscInt, is it the same
> as integer(8)? Or does it depend on the actual number?
>

PetscInt will be integer(4) if you configure with 32-bit ints, and
integer(8) if you configure with 64-bit ints. If you use it consistently,
you can avoid problems
with matching the PETSc API.

I wonder if I replace all my integer to PetscInt, will there be a large
> increase in memory usage, because all integer(4) now becomes integer(8)?
>

Only if you have large integer storage. Most codes do not.

  Thanks,

    Matt


> Thanks.
>
>
>   Thanks,
>
>      Matt
>
>
>>
>> --
>> Thank you very much.
>>
>> Yours sincerely,
>>
>> ================================================
>> TAY Wee-Beng (Zheng Weiming) 郑伟明
>> Personal research webpage: http://tayweebeng.wixsite.com/website
>> Youtube research showcase: https://www.youtube.com/channe
>> l/UC72ZHtvQNMpNs2uRTSToiLA
>> linkedin: www.linkedin.com/in/tay-weebeng
>> ================================================
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180220/cba0a157/attachment-0001.html>