<div dir="ltr">cc'ing petsc-dev.<div><br></div><div>I will try it.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 16, 2014 at 10:21 AM, Ed D'Azevedo <span dir="ltr"><<a href="mailto:dazevedoef@ornl.gov" target="_blank">dazevedoef@ornl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><br>
Hi Mark,<br>
<br>
I got an error when I tried the simple test code (see attached) on
Titan.<br>
<br>
Can you try to run the attached test case to see if it will work
for you?<br>
<br>
I have also sent the simple test code to Barry.<br>
<br>
<br>
<br>
The code seems to work with 1 thread<br>
<br>
env003> export OMP_NUM_THREADS=1<br>
env003> aprun -n 1 -d 16 ./tpetsc_madams<br>
PETSC_VERSION_RELEASE 0<br>
PETSC_VERSION_MAJOR 3<br>
PETSC_VERSION_MINOR 4<br>
PETSC_VERSION_SUBMINOR 4<br>
PETSC_VERSION_PATCH 0<br>
PETSC_VERSION_DATE unknown<br>
petsc_version_lt(3,3,0) is false<br>
nthreads = 1 NCASES = 100<br>
nz = 88804<br>
Warning: ieee_inexact is signaling<br>
all done<br>
total time is 7.268438 <br>
maxval(err) 6.9650285539069046E-011<br>
Application 4895013 resources: utime ~8s, stime ~0s, Rss ~22300,
inblocks ~11428, outblocks ~35848<br>
<br>
<br>
The code seems to have trouble using more threads<br>
<br>
env003> export OMP_NUM_THREADS=16<br>
env003> aprun -n 1 -d 16 ./tpetsc_madams<br>
*** glibc detected *** ./tpetsc_madams: double free or corruption
(!prev): 0x00000000015ee0b0 ***<br>
PETSC_VERSION_RELEASE 0<br>
PETSC_VERSION_MAJOR 3<br>
PETSC_VERSION_MINOR 4<br>
PETSC_VERSION_SUBMINOR 4<br>
PETSC_VERSION_PATCH 0<br>
PETSC_VERSION_DATE unknown<br>
petsc_version_lt(3,3,0) is false<br>
nthreads = 16 NCASES = 100<br>
tpetsc_madams: malloc.c:3091: sYSMALLOc: Assertion `(old_top ==
(((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) -
__builtin_offsetof (struct malloc_chunk, fd)))) &&
old_size == 0) || ((unsigned long) (old_size) >= (unsigned
long)((((__builtin_offsetof (struct malloc_chunk,
fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 *
(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1)
&& ((unsigned long)old_end & pagemask) == 0)' failed.<br>
Error: abort<br>
rax 0000000000000000, rbx 0000000000000fff, rcx
ffffffffffffffff<br>
rdx 0000000000000006, rsp 00002aaac4554f78, rbp
00002aaad0000098<br>
rsi 0000000000005200, rdi 00000000000051f1, r8
00000000ffffffff<br>
r9 00002aaaba8f9e40, r10 0000000000000008, r11
0000000000000202<br>
r12 0000000000000000, r13 00002aaad0008e30, r14
0000000000000020<br>
r15 0000000000000000<br>
======= Backtrace: =========<br>
/lib64/libc.so.6(+0x75558)[0x2aaaba5ff558]<br>
/lib64/libc.so.6(cfree+0x6c)[0x2aaaba6044fc]<br>
./tpetsc_madams[0x4b7759]<br>
/lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
/lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
/lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
/lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
/lib64/libc.so.6(+0x789f0) [0x2aaaba6029f0]<br>
/lib64/libc.so.6(__libc_malloc+0x77) [0x2aaaba6045e7]<br>
./tpetsc_madams() [0x4b76e3]<br>
./tpetsc_madams() [0x4aa3d7]<br>
./tpetsc_madams() [0x484888]<br>
./tpetsc_madams() [0x475798]<br>
./tpetsc_madams() [0x548aa5]<br>
./tpetsc_madams() [0x5e5df2]<br>
./tpetsc_madams() [0x57a353]<br>
./tpetsc_madams() [0x4a701e]<br>
./tpetsc_madams() [0x44e9be]<br>
./tpetsc_madams() [0x44e1f4]<br>
/lib64/libc.so.6(__libc_start_main+0xe6) [0x2aaaba5a8c36]<br>
./tpetsc_madams() [0x44e0e9]<br>
Application 4895021 exit codes: 127<br>
Application 4895021 resources: utime ~0s, stime ~0s, Rss ~12544,
inblocks ~11429, outblocks ~35849<div><div class="h5"><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On 04/11/2014 04:34 PM, Mark Adams wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">PETSc dev 'master' now has Barry's thread safe
stuff so you should be able to use that. I have build it in:
<div><br>
</div>
<div>
<div>PETSC_DIR=/autofs/na3_home1/adams/petsc</div>
<div>PETSC_ARCH=arch-titan-opt</div>
</div>
<div><br>
</div>
<div>So try this version out. And revert the code to the repo
version by doing:</div>
<div><br>
</div>
<div>> git checkout poisson.F90</div>
<div><br>
</div>
<div>and any other place where #if PETSC_VERSION_GE(3,5,0) is
used. I only see poisson.F90.</div>
<div><br>
</div>
<div>If this works I can install it wherever you like as Ed did.</div>
<div><br>
</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Thu, Apr 10, 2014 at 11:24 PM, Seung-Hoe Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>Hi Mark,<br>
<br>
I think it is not a big problem until we use petsc
3.5 other than you. Now I disabled #if #endif for
other users. Sorry for the inconvenience. Could you
uncomment it when you are working with 3.5, and does
not commit to core_dev, please?<br>
</div>
We need Ed's thread safe petsc for performance issue,
so we can find some way to resolve it when 3.5 is
released and installed on titan.<br>
<br>
</div>
Thanks,<br>
</div>
Seung-Hoe<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra">
<br>
<br>
<div class="gmail_quote">On Thu, Apr 10, 2014 at 11:38
AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">We've figured out the problem
(again). It will be fixed in future versions.
<div><br>
</div>
<div>We can fix your installation. I'm guessing
Ed did this installation so it might not be
worth fixing this code since you are up and
running. Let me know what you want to do. </div>
<div><br>
</div>
<div>The problem is that is a development
version of the code, which is not as stable as
the releases. This version needs to be
updated to get fix this problem. I have to
tell you how to do this update so let me know
if you want to do it.</div>
<span><font color="#888888">
<div><br>
</div>
<div>Mark</div>
<div>
<div>
<div><br>
</div>
</div>
</div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Apr 10,
2014 at 8:58 AM, Seung-Hoe Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>One question is..<br>
</div>
If PETSC_VERSION_GE is defined with
PETSC_VERSION_GT and
PETSC_VERSION_GT is defined with
something in petsc.h,<br>
</div>
redefinition of PETSC_VERSION_GT after
petsc.h will change PETSC_VERSION_GE?<br>
<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu,
Apr 10, 2014 at 10:55 AM,
Seung-Hoe Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">This is the
code I tried to compile.<br>
<div><br>
#if
PETSC_VERSION_GE(5,5,0)<br>
214
BBBBBBBBBBBBBBBBBBB=0<br>
215 call
KSPSetOperators(solver%ksp,
solver%Amat, solver%Amat,
ierr )<br>
216 #else<br>
217 AAAAAAAAAAAA=0<br>
218 call
KSPSetOperators(solver%ksp,
solver%Amat, solver%Amat,
SAME_NONZERO_PATT ERN,
ierr )<br>
219 #endif<br>
<br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Thu, Apr 10, 2014
at 10:49 AM, Mark
Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
<div>On Thu,
Apr 10, 2014
at 8:25 AM,
Seung-Hoe Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>I have
the same
problem. I
used
poisson.F90 in
/lustre/atlas2/env003/scratch/shku/XGC1_3_petsc_problem/
<br>
</div>
<br>
</div>
It seems that
#undef should
be after
#include<finclude/petsc.h>.
Otherwise,
petsc.h seems
to try
redefine it.<br>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div>Oh yes.</div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
Anyway, I got
the same error
message:
<div><br>
PGF90-S-0038-Symbol,
bbbbbbbbbbbbbbbbbbb,
has not been
explicitly
declared
(poisson.F90)<br>
0 inform,
0 warnings,
1 severes, 0
fatal for
init_1field_solver<br>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div>is this
in the source
file? </div>
<div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
</div>
</div>
I tried 5.5.0
for the
arguments of
PETSC_VERSION_GE,
but still have
the same
problem.<br>
<br>
</div>
Or do you mean
VERSION_GE and
VERSION_LT
instead of
VERSION_GT and
VERSION_LE?<br>
<br>
</div>
<div>Thanks,<br>
</div>
Seung-Hoe<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 9:54
PM, Mark Adams
<span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">We
might have a
fix. It turns
out that some
fortran
compilers to
not do the
#define quite
right. Try
this:
<div>
<br>
</div>
<div>> git
checkout
poisson.F90</div>
<div><br>
</div>
<div>This will
out the
original (bad)
file back.
Then add to
poisson.F90</div>
<div>
<div><br>
</div>
<div>#undef
PETSC_VERSION_GT</div>
<div>
<div>#define
PETSC_VERSION_GT(MAJOR,MINOR,SUBMINOR)
\</div>
<div>
(0==PETSC_VERSION_LE(MAJOR,MINOR,SUBMINOR))</div>
<div><br>
</div>
</div>
</div>
<div>This is
the fix that
Jed thinks
will work. If
it works we
can propagate
it.</div>
<div><br>
</div>
<div>Sorry
about the
confusion,</div>
<div>Mark</div>
<div><br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 7:36
PM, Seung-Hoe
Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Great!
Thank you.<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Wed, Apr 9,
2014 at 9:35
PM, Mark Adams
<span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Good
news. I'm at
a meeting and
Jed Brown is
here and he
seems to have
been able to
reproduce the
error.
Preprocessors
can be a pain.
I'm going to
wait until Jed
has a chance
to look at
this and come
up with a
solution.</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 7:28
PM, Seung-Hoe
Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Now I am
using
iterative
solver which
will be
replaced to
your 2 field
solver later.<br>
Sorry. The
line number
3003 is wrong.
It is 209 or
near it.<br>
</div>
<div>Yes. We
can take a
look at this
when you come.
It is not
urgent
problem.<br>
<br>
</div>
<div>Thanks,<br>
Seung-Hoe<br>
<br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 9:25
PM, Mark Adams
<span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I
noticed that
your makefile
was not using
the new petsc
solver. Is
that
intensional?
<div><br>
</div>
<div>
The line you
gave me 3003
poission.F90
seems to be
the last line
of the file.</div>
<div><br>
</div>
<div>
I am not
getting Edison
or Titan to
build. We can
take a look at
this next
week.</div>
<span><font color="#888888">
<div><br>
</div>
</font></span>
<div><span><font color="#888888">Mark</font></span>
<div>
<div><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Tue, Apr 8,
2014 at 12:38
PM, Seung-Hoe
Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>Hi Mark,<br>
<br>
</div>
It seems that
PETSC_VERSION_GE
is not working
line 3003 of
poisson.F90.
Is there any
reason of
using
PETSC_VERSION_GE
instead of
PETSC_VERSION_LE?
The previous
code worked, I
think.<br>
<br>
</div>
Thanks,<br>
Seung-Hoe<br>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div></div>