<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Feb 20, 2018 at 8:08 PM, TAY wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<br>
<div class="m_-9144408435498121880moz-cite-prefix">On 21/2/2018 9:00 AM, Matthew Knepley
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Feb 20, 2018 at 7:54 PM, TAY
wee-beng <span dir="ltr"><<a href="mailto:zonexo@gmail.com" target="_blank">zonexo@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
When I run my CFD code with a grid size of 1119x1119x499 (
total grid size = 624828339 ), I got the error saying I
need to compile PETSc with 64-bit indices.<br>
<br>
So I tried to compile PETSc again and then compile my CFD
code with the newly compiled PETSc. However, now I got
segmentation error:<br>
<br>
rm: cannot remove `log': No such file or directory<br>
[409]PETSC ERROR: ------------------------------<wbr>------------------------------<wbr>------------<br>
[409]PETSC ERROR: [535]PETSC ERROR: [410]PETSC ERROR:
------------------------------<wbr>------------------------------<wbr>------------<br>
[410]PETSC ERROR: Caught signal number 11 SEGV:
Segmentation Violation, probably memory access out of
range<br>
[410]PETSC ERROR: Try option -start_in_debugger or
-on_error_attach_debugger<br>
[410]PETSC ERROR: [536]PETSC ERROR:
------------------------------<wbr>------------------------------<wbr>------------<br>
[536]PETSC ERROR: Caught signal number 11 SEGV:
Segmentation Violation, probably memory access out of
range<br>
[536]PETSC ERROR: Try option -start_in_debugger or
-on_error_attach_debugger<br>
[536]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
[536]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to find memory corruption
errors<br>
[536]PETSC ERROR: likely location of problem given in
stack below<br>
[536]PETSC ERROR: --------------------- Stack Frames
------------------------------<wbr>------<br>
[536]PETSC ERROR: Note: The EXACT line numbers in the
stack are not available,<br>
[536]PETSC ERROR: INSTEAD the line number of the
start of the function<br>
[536]PETSC ERROR: is given.<br>
[536]PETSC ERROR: [536] DMDACheckOwnershipRanges_Priva<wbr>te
line 581 /home/users/nus/tsltaywb/sourc<wbr>e/petsc-3.7.6/src/dm/impls/da/<wbr>da.c<br>
[536]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/d<wbr>ocumentation/faq.html#valgrind</a><br>
[410]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a>
on GNU/linux and Apple Mac OS X to find memory corruption
errors<br>
[410]PETSC ERROR: likely location of problem given in
stack below<br>
[410]PETSC ERROR: --------------------- Stack Frames
------------------------------<wbr>------<br>
[410]PETSC ERROR: Note: The EXACT line numbers in the
stack are not available,<br>
[897]PETSC ERROR: [536] DMDASetOwnershipRanges line 613
/home/users/nus/tsltaywb/sourc<wbr>e/petsc-3.7.6/src/dm/impls/da/<wbr>da.c<br>
[536]PETSC ERROR: [536] DMDACreate3d line 1434
/home/users/nus/tsltaywb/sourc<wbr>e/petsc-3.7.6/src/dm/impls/da/<wbr>da3.c<br>
[536]PETSC ERROR: --------------------- Error Message
------------------------------<wbr>------------------------------<wbr>--<br>
<br>
The CFD code worked previously but increasing the problem
size results in segmentation error. It seems to be related
to DMDACreate3d and DMDASetOwnershipRanges. Any idea where
the problem lies?<br>
<br>
Besides, I want to know when and why do I have to use
PETSc with 64-bit indices?<br>
</blockquote>
<div><br>
</div>
<div>1) A 32-bit integer can hold numbers up to 2^32 =
4.2e9, so if you have a 3D velocity, pressure, and energy,
you already have 3e9 unknowns,</div>
<div> before you even start to count nonzero entries in
the matrix. 64-bit integers allow you to handle these big
sizes.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Also, can I use the 64-bit indices version with smaller
sized problems?<br>
</blockquote>
<div><br>
</div>
<div>2) Yes</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
And is there a speed difference between using the 32-bit
and 64-bit indices ver?</blockquote>
<div><br>
</div>
<div>3) I have seen no evidence of this</div>
<div><br>
</div>
<div>4) My guess is that you have defines regular integers
in your code and passed them to PETSc, rather than using
PetscInt as the type.</div>
</div>
</div>
</div>
</blockquote>
Oh that seems probable. So I am still using integer(4) when it
should be integer(8) for some values, is that so? If I use PetscInt,
is it the same as integer(8)? Or does it depend on the actual
number?<br></div></blockquote><div><br></div><div>PetscInt will be integer(4) if you configure with 32-bit ints, and integer(8) if you configure with 64-bit ints. If you use it consistently, you can avoid problems</div><div>with matching the PETSc API.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
I wonder if I replace all my integer to PetscInt, will there be a
large increase in memory usage, because all integer(4) now becomes
integer(8)?<br></div></blockquote><div><br></div><div>Only if you have large integer storage. Most codes do not.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
Thanks.<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="m_-9144408435498121880HOEnZb"><font color="#888888"><br>
-- <br>
Thank you very much.<br>
<br>
Yours sincerely,<br>
<br>
==============================<wbr>==================<br>
TAY Wee-Beng (Zheng Weiming) 郑伟明<br>
Personal research webpage: <a href="http://tayweebeng.wixsite.com/website" rel="noreferrer" target="_blank">http://tayweebeng.wixsite.com/<wbr>website</a><br>
Youtube research showcase: <a href="https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA" rel="noreferrer" target="_blank">https://www.youtube.com/channe<wbr>l/UC72ZHtvQNMpNs2uRTSToiLA</a><br>
linkedin: <a href="http://www.linkedin.com/in/tay-weebeng" rel="noreferrer" target="_blank">www.linkedin.com/in/tay-weeben<wbr>g</a><br>
==============================<wbr>==================<br>
<br>
</font></span></blockquote>
</div>
<br>
<br clear="all"><span class="HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_-9144408435498121880gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~<wbr>knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>