<div dir="ltr">I could also use your compile line. I am getting no output.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 16, 2014 at 10:41 AM, Ed D'Azevedo <span dir="ltr"><<a href="mailto:dazevedoef@ornl.gov" target="_blank">dazevedoef@ornl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>Hi Mark,<br>
<br>
I hope this back trace might be helpful.<br>
<br>
You may need to build petsc with <br>
./configure \
<br>
--with-x=0 \
<br>
--with-debugging=0 \
<br>
--with-log=0 \
<br>
<br>
<br>
env003> addr2line --exe=tpetsc_madams 0x4b76e3 0x4aa3d7
0x484888 0x475798 0x548aa5 0x5e5df2 0x57a353 0x4a701e 0x44e9be
0x44e1f4 <br>
/autofs/na3_home1/adams/petsc/src/sys/memory/mal.c:27<br>
/autofs/na3_home1/adams/petsc/src/sys/utils/str.c:188<br>
/autofs/na3_home1/adams/petsc/src/sys/logging/utils/eventlog.c:317<br>
/autofs/na3_home1/adams/petsc/src/sys/logging/plog.c:747<br>
/autofs/na3_home1/adams/petsc/src/mat/interface/dlregismat.c:145<br>
/autofs/na3_home1/adams/petsc/src/mat/utils/gcreate.c:57<br>
/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3576<br>
/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/ftn-custom/zaijf.c:14<br>
/autofs/na3_home1/efdazedo/test/PETSC/./tpetsc.F90:147<div class=""><br>
<br>
<br>
<br>
<br>
======= Backtrace: =========<br>
/lib64/libc.so.6(+0x75558)[0x2aaaba5ff558]<br>
/lib64/libc.so.6(cfree+0x6c)[0x2aaaba6044fc]<br>
./tpetsc_madams[0x4b7759]<br>
/lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
/lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
/lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
/lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
/lib64/libc.so.6(+0x789f0) [0x2aaaba6029f0]<br>
/lib64/libc.so.6(__libc_malloc+0x77) [0x2aaaba6045e7]<br>
./tpetsc_madams() [0x4b76e3]<br>
./tpetsc_madams() [0x4aa3d7]<br>
./tpetsc_madams() [0x484888]<br>
./tpetsc_madams() [0x475798]<br>
./tpetsc_madams() [0x548aa5]<br>
./tpetsc_madams() [0x5e5df2]<br>
./tpetsc_madams() [0x57a353]<br>
./tpetsc_madams() [0x4a701e]<br>
./tpetsc_madams() [0x44e9be]<br>
./tpetsc_madams() [0x44e1f4]<br>
/lib64/libc.so.6(__libc_start_main+0xe6) [0x2aaaba5a8c36]<br>
./tpetsc_madams() [0x44e0e9]<br>
<br>
<br>
<br>
<br>
<br>
<br></div><div><div class="h5">
On 04/16/2014 10:33 AM, Mark Adams wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr">cc'ing petsc-dev.
<div><br>
</div>
<div>I will try it.</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Apr 16, 2014 at 10:21 AM, Ed
D'Azevedo <span dir="ltr"><<a href="mailto:dazevedoef@ornl.gov" target="_blank">dazevedoef@ornl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div><br>
Hi Mark,<br>
<br>
I got an error when I tried the simple test code (see
attached) on Titan.<br>
<br>
Can you try to run the attached test case to see if it
will work for you?<br>
<br>
I have also sent the simple test code to Barry.<br>
<br>
<br>
<br>
The code seems to work with 1 thread<br>
<br>
env003> export OMP_NUM_THREADS=1<br>
env003> aprun -n 1 -d 16 ./tpetsc_madams<br>
PETSC_VERSION_RELEASE 0<br>
PETSC_VERSION_MAJOR 3<br>
PETSC_VERSION_MINOR 4<br>
PETSC_VERSION_SUBMINOR 4<br>
PETSC_VERSION_PATCH 0<br>
PETSC_VERSION_DATE unknown<br>
petsc_version_lt(3,3,0) is false<br>
nthreads = 1 NCASES = 100<br>
nz = 88804<br>
Warning: ieee_inexact is signaling<br>
all done<br>
total time is 7.268438 <br>
maxval(err) 6.9650285539069046E-011<br>
Application 4895013 resources: utime ~8s, stime ~0s,
Rss ~22300, inblocks ~11428, outblocks ~35848<br>
<br>
<br>
The code seems to have trouble using more threads<br>
<br>
env003> export OMP_NUM_THREADS=16<br>
env003> aprun -n 1 -d 16 ./tpetsc_madams<br>
*** glibc detected *** ./tpetsc_madams: double free or
corruption (!prev): 0x00000000015ee0b0 ***<br>
PETSC_VERSION_RELEASE 0<br>
PETSC_VERSION_MAJOR 3<br>
PETSC_VERSION_MINOR 4<br>
PETSC_VERSION_SUBMINOR 4<br>
PETSC_VERSION_PATCH 0<br>
PETSC_VERSION_DATE unknown<br>
petsc_version_lt(3,3,0) is false<br>
nthreads = 16 NCASES = 100<br>
tpetsc_madams: malloc.c:3091: sYSMALLOc: Assertion
`(old_top == (((mbinptr) (((char *)
&((av)->bins[((1) - 1) * 2])) -
__builtin_offsetof (struct malloc_chunk, fd))))
&& old_size == 0) || ((unsigned long)
(old_size) >= (unsigned long)((((__builtin_offsetof
(struct malloc_chunk, fd_nextsize))+((2 *
(sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t)))
- 1))) && ((old_top)->size & 0x1)
&& ((unsigned long)old_end & pagemask) ==
0)' failed.<br>
Error: abort<br>
rax 0000000000000000, rbx 0000000000000fff, rcx
ffffffffffffffff<br>
rdx 0000000000000006, rsp 00002aaac4554f78, rbp
00002aaad0000098<br>
rsi 0000000000005200, rdi 00000000000051f1, r8
00000000ffffffff<br>
r9 00002aaaba8f9e40, r10 0000000000000008, r11
0000000000000202<br>
r12 0000000000000000, r13 00002aaad0008e30, r14
0000000000000020<br>
r15 0000000000000000<br>
======= Backtrace: =========<br>
/lib64/libc.so.6(+0x75558)[0x2aaaba5ff558]<br>
/lib64/libc.so.6(cfree+0x6c)[0x2aaaba6044fc]<br>
./tpetsc_madams[0x4b7759]<br>
/lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
/lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
/lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
/lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
/lib64/libc.so.6(+0x789f0) [0x2aaaba6029f0]<br>
/lib64/libc.so.6(__libc_malloc+0x77)
[0x2aaaba6045e7]<br>
./tpetsc_madams() [0x4b76e3]<br>
./tpetsc_madams() [0x4aa3d7]<br>
./tpetsc_madams() [0x484888]<br>
./tpetsc_madams() [0x475798]<br>
./tpetsc_madams() [0x548aa5]<br>
./tpetsc_madams() [0x5e5df2]<br>
./tpetsc_madams() [0x57a353]<br>
./tpetsc_madams() [0x4a701e]<br>
./tpetsc_madams() [0x44e9be]<br>
./tpetsc_madams() [0x44e1f4]<br>
/lib64/libc.so.6(__libc_start_main+0xe6)
[0x2aaaba5a8c36]<br>
./tpetsc_madams() [0x44e0e9]<br>
Application 4895021 exit codes: 127<br>
Application 4895021 resources: utime ~0s, stime ~0s,
Rss ~12544, inblocks ~11429, outblocks ~35849
<div>
<div><br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
On 04/11/2014 04:34 PM, Mark Adams wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">PETSc dev 'master' now has Barry's
thread safe stuff so you should be able to use
that. I have build it in:
<div><br>
</div>
<div>
<div>PETSC_DIR=/autofs/na3_home1/adams/petsc</div>
<div>PETSC_ARCH=arch-titan-opt</div>
</div>
<div><br>
</div>
<div>So try this version out. And revert the
code to the repo version by doing:</div>
<div><br>
</div>
<div>> git checkout poisson.F90</div>
<div><br>
</div>
<div>and any other place where #if
PETSC_VERSION_GE(3,5,0) is used. I only see
poisson.F90.</div>
<div><br>
</div>
<div>If this works I can install it wherever you
like as Ed did.</div>
<div><br>
</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote"> On Thu, Apr 10, 2014
at 11:24 PM, Seung-Hoe Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>Hi Mark,<br>
<br>
I think it is not a big problem
until we use petsc 3.5 other than
you. Now I disabled #if #endif for
other users. Sorry for the
inconvenience. Could you uncomment
it when you are working with 3.5,
and does not commit to core_dev,
please?<br>
</div>
We need Ed's thread safe petsc for
performance issue, so we can find some
way to resolve it when 3.5 is released
and installed on titan.<br>
<br>
</div>
Thanks,<br>
</div>
Seung-Hoe<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">On Thu, Apr
10, 2014 at 11:38 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">We've figured out
the problem (again). It will be
fixed in future versions.
<div><br>
</div>
<div>We can fix your
installation. I'm guessing Ed
did this installation so it
might not be worth fixing this
code since you are up and
running. Let me know what you
want to do. </div>
<div><br>
</div>
<div>The problem is that is a
development version of the
code, which is not as stable
as the releases. This version
needs to be updated to get fix
this problem. I have to tell
you how to do this update so
let me know if you want to do
it.</div>
<span><font color="#888888">
<div><br>
</div>
<div>Mark</div>
<div>
<div>
<div><br>
</div>
</div>
</div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Thu, Apr 10, 2014 at 8:58
AM, Seung-Hoe Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>One question
is..<br>
</div>
If PETSC_VERSION_GE
is defined with
PETSC_VERSION_GT and
PETSC_VERSION_GT is
defined with
something in
petsc.h,<br>
</div>
redefinition of
PETSC_VERSION_GT after
petsc.h will change
PETSC_VERSION_GE?<br>
<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Thu, Apr 10,
2014 at 10:55
AM, Seung-Hoe Ku
<span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">This
is the code I
tried to
compile.<br>
<div><br>
#if
PETSC_VERSION_GE(5,5,0)<br>
214
BBBBBBBBBBBBBBBBBBB=0<br>
215 call
KSPSetOperators(solver%ksp,
solver%Amat,
solver%Amat,
ierr )<br>
216 #else<br>
217
AAAAAAAAAAAA=0<br>
218 call
KSPSetOperators(solver%ksp,
solver%Amat,
solver%Amat,
SAME_NONZERO_PATT
ERN, ierr )<br>
219 #endif<br>
<br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Thu, Apr
10, 2014 at
10:49 AM, Mark
Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
<div>On Thu,
Apr 10, 2014
at 8:25 AM,
Seung-Hoe Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>I have
the same
problem. I
used
poisson.F90 in
/lustre/atlas2/env003/scratch/shku/XGC1_3_petsc_problem/
<br>
</div>
<br>
</div>
It seems that
#undef should
be after
#include<finclude/petsc.h>.
Otherwise,
petsc.h seems
to try
redefine it.<br>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div>Oh yes.</div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div> Anyway,
I got the same
error message:
<div><br>
PGF90-S-0038-Symbol,
bbbbbbbbbbbbbbbbbbb,
has not been
explicitly
declared
(poisson.F90)<br>
0 inform,
0 warnings,
1 severes, 0
fatal for
init_1field_solver<br>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div>is this
in the source
file? </div>
<div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div> </div>
</div>
I tried 5.5.0
for the
arguments of
PETSC_VERSION_GE,
but still have
the same
problem.<br>
<br>
</div>
Or do you mean
VERSION_GE and
VERSION_LT
instead of
VERSION_GT and
VERSION_LE?<br>
<br>
</div>
<div>Thanks,<br>
</div>
Seung-Hoe<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 9:54
PM, Mark Adams
<span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">We
might have a
fix. It turns
out that some
fortran
compilers to
not do the
#define quite
right. Try
this:
<div> <br>
</div>
<div>> git
checkout
poisson.F90</div>
<div><br>
</div>
<div>This will
out the
original (bad)
file back.
Then add to
poisson.F90</div>
<div>
<div><br>
</div>
<div>#undef
PETSC_VERSION_GT</div>
<div>
<div>#define
PETSC_VERSION_GT(MAJOR,MINOR,SUBMINOR)
\</div>
<div>
(0==PETSC_VERSION_LE(MAJOR,MINOR,SUBMINOR))</div>
<div><br>
</div>
</div>
</div>
<div>This is
the fix that
Jed thinks
will work. If
it works we
can propagate
it.</div>
<div><br>
</div>
<div>Sorry
about the
confusion,</div>
<div>Mark</div>
<div><br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 7:36
PM, Seung-Hoe
Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Great!
Thank you.<br>
<br>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Wed, Apr 9,
2014 at 9:35
PM, Mark Adams
<span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Good
news. I'm at
a meeting and
Jed Brown is
here and he
seems to have
been able to
reproduce the
error.
Preprocessors
can be a pain.
I'm going to
wait until Jed
has a chance
to look at
this and come
up with a
solution.</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 7:28
PM, Seung-Hoe
Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Now I am
using
iterative
solver which
will be
replaced to
your 2 field
solver later.<br>
Sorry. The
line number
3003 is wrong.
It is 209 or
near it.<br>
</div>
<div>Yes. We
can take a
look at this
when you come.
It is not
urgent
problem.<br>
<br>
</div>
<div>Thanks,<br>
Seung-Hoe<br>
<br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Apr 9,
2014 at 9:25
PM, Mark Adams
<span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I
noticed that
your makefile
was not using
the new petsc
solver. Is
that
intensional?
<div><br>
</div>
<div> The line
you gave me
3003
poission.F90
seems to be
the last line
of the file.</div>
<div><br>
</div>
<div> I am not
getting Edison
or Titan to
build. We can
take a look at
this next
week.</div>
<span><font color="#888888">
<div><br>
</div>
</font></span>
<div><span><font color="#888888">Mark</font></span>
<div>
<div><br>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Tue, Apr 8,
2014 at 12:38
PM, Seung-Hoe
Ku <span dir="ltr"><<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>Hi Mark,<br>
<br>
</div>
It seems that
PETSC_VERSION_GE
is not working
line 3003 of
poisson.F90.
Is there any
reason of
using
PETSC_VERSION_GE
instead of
PETSC_VERSION_LE?
The previous
code worked, I
think.<br>
<br>
</div>
Thanks,<br>
Seung-Hoe<br>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>