<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 21/2/2018 9:12 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMYG4Gmkq6RmfdfqhCVCg4JSacGiU-UmxHjhfMdvpdArA60+FQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Feb 20, 2018 at 8:08 PM, TAY
wee-beng <span dir="ltr"><<a
href="mailto:zonexo@gmail.com" target="_blank"
moz-do-not-send="true">zonexo@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<div class="m_-9144408435498121880moz-cite-prefix">On
21/2/2018 9:00 AM, Matthew Knepley wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Feb 20, 2018 at
7:54 PM, TAY wee-beng <span dir="ltr"><<a
href="mailto:zonexo@gmail.com"
target="_blank" moz-do-not-send="true">zonexo@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">Hi,<br>
<br>
When I run my CFD code with a grid size of
1119x1119x499 ( total grid size = 624828339
), I got the error saying I need to compile
PETSc with 64-bit indices.<br>
<br>
So I tried to compile PETSc again and then
compile my CFD code with the newly compiled
PETSc. However, now I got segmentation error:<br>
<br>
rm: cannot remove `log': No such file or
directory<br>
[409]PETSC ERROR:
------------------------------<wbr>------------------------------<wbr>------------<br>
[409]PETSC ERROR: [535]PETSC ERROR: [410]PETSC
ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
[410]PETSC ERROR: Caught signal number 11
SEGV: Segmentation Violation, probably memory
access out of range<br>
[410]PETSC ERROR: Try option
-start_in_debugger or
-on_error_attach_debugger<br>
[410]PETSC ERROR: [536]PETSC ERROR:
------------------------------<wbr>------------------------------<wbr>------------<br>
[536]PETSC ERROR: Caught signal number 11
SEGV: Segmentation Violation, probably memory
access out of range<br>
[536]PETSC ERROR: Try option
-start_in_debugger or
-on_error_attach_debugger<br>
[536]PETSC ERROR: or see <a
href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
[536]PETSC ERROR: or try <a
href="http://valgrind.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to find memory
corruption errors<br>
[536]PETSC ERROR: likely location of problem
given in stack below<br>
[536]PETSC ERROR: --------------------- Stack
Frames ------------------------------<wbr>------<br>
[536]PETSC ERROR: Note: The EXACT line numbers
in the stack are not available,<br>
[536]PETSC ERROR: INSTEAD the line
number of the start of the function<br>
[536]PETSC ERROR: is given.<br>
[536]PETSC ERROR: [536]
DMDACheckOwnershipRanges_Priva<wbr>te line 581
/home/users/nus/tsltaywb/sourc<wbr>e/petsc-3.7.6/src/dm/impls/da/<wbr>da.c<br>
[536]PETSC ERROR: or see <a
href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
[410]PETSC ERROR: or try <a
href="http://valgrind.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to find memory
corruption errors<br>
[410]PETSC ERROR: likely location of problem
given in stack below<br>
[410]PETSC ERROR: --------------------- Stack
Frames ------------------------------<wbr>------<br>
[410]PETSC ERROR: Note: The EXACT line numbers
in the stack are not available,<br>
[897]PETSC ERROR: [536] DMDASetOwnershipRanges
line 613 /home/users/nus/tsltaywb/sourc<wbr>e/petsc-3.7.6/src/dm/impls/da/<wbr>da.c<br>
[536]PETSC ERROR: [536] DMDACreate3d line 1434
/home/users/nus/tsltaywb/sourc<wbr>e/petsc-3.7.6/src/dm/impls/da/<wbr>da3.c<br>
[536]PETSC ERROR: --------------------- Error
Message ------------------------------<wbr>------------------------------<wbr>--<br>
<br>
The CFD code worked previously but increasing
the problem size results in segmentation
error. It seems to be related to DMDACreate3d
and DMDASetOwnershipRanges. Any idea where the
problem lies?<br>
<br>
Besides, I want to know when and why do I have
to use PETSc with 64-bit indices?<br>
</blockquote>
<div><br>
</div>
<div>1) A 32-bit integer can hold numbers up to
2^32 = 4.2e9, so if you have a 3D velocity,
pressure, and energy, you already have 3e9
unknowns,</div>
<div> before you even start to count nonzero
entries in the matrix. 64-bit integers allow
you to handle these big sizes.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"> Also, can I use the
64-bit indices version with smaller sized
problems?<br>
</blockquote>
<div><br>
</div>
<div>2) Yes</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"> And is there a speed
difference between using the 32-bit and 64-bit
indices ver?</blockquote>
<div><br>
</div>
<div>3) I have seen no evidence of this</div>
<div><br>
</div>
<div>4) My guess is that you have defines
regular integers in your code and passed them
to PETSc, rather than using PetscInt as the
type.</div>
</div>
</div>
</div>
</blockquote>
Oh that seems probable. So I am still using integer(4)
when it should be integer(8) for some values, is that
so? If I use PetscInt, is it the same as integer(8)? Or
does it depend on the actual number?<br>
</div>
</blockquote>
<div><br>
</div>
<div>PetscInt will be integer(4) if you configure with
32-bit ints, and integer(8) if you configure with 64-bit
ints. If you use it consistently, you can avoid problems</div>
<div>with matching the PETSc API.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> I wonder if I
replace all my integer to PetscInt, will there be a
large increase in memory usage, because all integer(4)
now becomes integer(8)?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Only if you have large integer storage. Most codes do
not.</div>
</div>
</div>
</div>
</blockquote>
Hi,<br>
<br>
What do you mean by "large integer storage"?<br>
<br>
Btw, I got the following error when I ran a simple small test case
with my CFD code:<br>
<br>
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[0]PETSC ERROR: Out of memory. This could be due to allocating<br>
[0]PETSC ERROR: too large an object or bleeding by not properly<br>
[0]PETSC ERROR: destroying unneeded objects.<br>
[0]PETSC ERROR: Memory allocated 0 Memory used by process 52858880<br>
[0]PETSC ERROR: Try running with -malloc_dump or -malloc_log for
info.<br>
[0]PETSC ERROR: Memory requested 6917565139726106624<br>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble
shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017<br>
[0]PETSC ERROR: ./a.out on a petsc-3.8.3_intel_64_rel named nus02 by
tsltaywb Thu Feb 22 10:34:29 2018<br>
[0]PETSC ERROR: Configure options
--with-mpi-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64
--with-blaslapack-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64
--download-hypre=/home/users/nus/tsltaywb/source/git.hypre.tar.gz
--with-debugging=0
--prefix=/home/users/nus/tsltaywb/lib/petsc-3.8.3_intel_64_rel
--with-shared-libraries=0 --known-mpi-shared-libraries=0
--with-fortran-interfaces=1 --CFLAGS="-xHost -g -O3"
--CXXFLAGS="-xHost -g -O3" --FFLAGS="-xHost -g -O3"
--with-64-bit-indices<br>
[0]PETSC ERROR: #105 DMSetUp_DA() line 18 in
/home/users/nus/tsltaywb/source/petsc-3.8.3/src/dm/impls/da/dareg.c<br>
[0]PETSC ERROR: #106 DMSetUp_DA() line 18 in
/home/users/nus/tsltaywb/source/petsc-3.8.3/src/dm/impls/da/dareg.c<br>
[0]PETSC ERROR: #107 DMSetUp() line 720 in
/home/users/nus/tsltaywb/source/petsc-3.8.3/src/dm/interface/dm.c<br>
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[0]PETSC ERROR: Arguments are incompatible<br>
[0]PETSC ERROR: Ownership ranges sum to 4294967337 but global
dimension is 41<br>
[0]PETSC ERROR: See
<a class="moz-txt-link-freetext" href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble
shooting.<br>
[0]PETSC ERROR: Petsc Release Version 3.8.3, Dec, 09, 2017<br>
[0]PETSC ERROR: ./a.out on a petsc-3.8.3_intel_64_rel named nus02 by
tsltaywb Thu Feb 22 10:34:29 2018<br>
[0]PETSC ERROR: Configure options
--with-mpi-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mpi/intel64
--with-blaslapack-dir=/app/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64
--download-hypre=/home/users/nus/tsltaywb/source/git.hypre.tar.gz
--with-debugging=0
--prefix=/home/users/nus/tsltaywb/lib/petsc-3.8.3_intel_64_rel
--with-shared-libraries=0 --known-mpi-shared-libraries=0
--with-fortran-interfaces=1 --CFLAGS="-xHost -g -O3"
--CXXFLAGS="-xHost -g -O3" --FFLAGS="-xHost -g -O3"
--with-64-bit-indices<br>
[0]PETSC ERROR: #108 DMDACheckOwnershipRanges_Private() line 548 in
/home/users/nus/tsltaywb/source/petsc-3.8.3/src/dm/impls/da/da.c<br>
[0]PETSC ERROR: #109 DMDASetOwnershipRanges() line 580 in
/home/users/nus/tsltaywb/source/petsc-3.8.3/src/dm/impls/da/da.c<br>
[0]PETSC ERROR: #110 DMDACreate3d() line 1444 in
/home/users/nus/tsltaywb/source/petsc-3.8.3/src/dm/impls/da/da3.c<br>
<br>
What's the problem?<br>
<br>
Thanks.<br>
<br>
<blockquote type="cite"
cite="mid:CAMYG4Gmkq6RmfdfqhCVCg4JSacGiU-UmxHjhfMdvpdArA60+FQ@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> Thanks.<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"><span
class="m_-9144408435498121880HOEnZb"><font
color="#888888"><br>
-- <br>
Thank you very much.<br>
<br>
Yours sincerely,<br>
<br>
==============================<wbr>==================<br>
TAY Wee-Beng (Zheng Weiming) 郑伟明<br>
Personal research webpage: <a
href="http://tayweebeng.wixsite.com/website"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://tayweebeng.wixsite.com/<wbr>website</a><br>
Youtube research showcase: <a
href="https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://www.youtube.com/channe<wbr>l/UC72ZHtvQNMpNs2uRTSToiLA</a><br>
linkedin: <a
href="http://www.linkedin.com/in/tay-weebeng"
rel="noreferrer" target="_blank"
moz-do-not-send="true">www.linkedin.com/in/tay-weeben<wbr>g</a><br>
==============================<wbr>==================<br>
<br>
</font></span></blockquote>
</div>
<br>
<br clear="all">
<span class="HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div
class="m_-9144408435498121880gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to
which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a
href="http://www.caam.rice.edu/%7Emk51/"
target="_blank"
moz-do-not-send="true">https://www.cse.buffalo.edu/~<wbr>knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/"
target="_blank" moz-do-not-send="true">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>