<div dir="ltr">Ed, you can try this on Titan:<div><br></div><div><div>> module swap PrgEnv-pgi PrgEnv-intel</div><div>> make PETSC_DIR=/autofs/na3_home1/adams/petsc PETSC_ARCH=arch-titan-opt-intel<br></div></div></div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 16, 2014 at 5:46 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class=""><br>
On Apr 16, 2014, at 3:46 PM, Ed D'Azevedo <<a href="mailto:dazevedoef@ornl.gov">dazevedoef@ornl.gov</a>> wrote:<br>
<br>
> Hi Barry,<br>
><br>
> I find that if I launch it multiple times, it may crash at different locations where it may not produce a stack trace.<br>
<br>
</div> Ed,<br>
<br>
This is the issue I was seeing with the C code that did not consistently even on my machine where your code works fine.<br>
<br>
So it may be your luck on this machine has failed until we get a proper support for this particular use of threads in PETSc which I hope will be by the end of the summer.<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> I tried to capture one case where it produced a stack trace.<br>
><br>
> Ed<br>
><br>
><br>
> On 04/16/2014 04:22 PM, Barry Smith wrote:<br>
>> This is puzzling. At one place it has<br>
>><br>
>>> Assertion failed in file /notbackedup/tmp/ulib/mpt/nightly/6.2/111913/mpich2/src/mpi/attr/comm_delete_attr.c at line 68: MPIU_Object_get_ref((p->keyval)) >= 0<br>
>>> *** glibc detected *** ./tpetsc_madams: double free or corruption (!prev): 0x000000000164e0c0 ***<br>
>> which is seems to indicate the MPI implementation is messing up by freeing the same thing twice (which it should not do if the MPI is thread safe)<br>
>><br>
>><br>
>> but then there seems to be an issue in the PETSc allocation.<br>
>><br>
>>> env003> addr2line --exe=tpetsc_madams 0x4b76d3 0x57a915 0x57a48e 0x4a700e 0x44e9be 0x44e1f4<br>
>>> /autofs/na3_home1/adams/petsc/src/sys/memory/mal.c:27<br>
>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3682<br>
>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3578<br>
>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/ftn-custom/zaijf.c:14<br>
>> Is there another version of MPI you can build with?<br>
>><br>
>> Barry<br>
>><br>
>><br>
>> On Apr 16, 2014, at 2:38 PM, Ed D'Azevedo <<a href="mailto:dazevedoef@ornl.gov">dazevedoef@ornl.gov</a>> wrote:<br>
>><br>
>>> Hi Mark,<br>
>>><br>
>>> I used the modified tpetsc.F90 in your email but I get the following error on Titan with 16 threads (works with 1 thread) with your version of petsc for titan.<br>
>>><br>
>>> PETSC_DIR=/autofs/na3_home1/adams/petsc<br>
>>> PETSC_ARCH=arch-titan-opt<br>
>>><br>
>>><br>
>>><br>
>>> aprun -n 1 -d 16 ./tpetsc_madams<br>
>>> Assertion failed in file /notbackedup/tmp/ulib/mpt/nightly/6.2/111913/mpich2/src/mpi/attr/comm_delete_attr.c at line 68: MPIU_Object_get_ref((p->keyval)) >= 0<br>
>>> *** glibc detected *** ./tpetsc_madams: double free or corruption (!prev): 0x000000000164e0c0 ***<br>
>>> Rank 0 [Wed Apr 16 15:30:53 2014] [c4-1c0s0n2] internal ABORT - process 0<br>
>>> Error: abort<br>
>>> rax 0000000000000000, rbx 00002aaabd5466c0, rcx ffffffffffffffff<br>
>>> rdx 0000000000000006, rsp 00002aaabd546538, rbp 0000000000000001<br>
>>> rsi 0000000000004ad9, rdi 0000000000004ad8, r8 00000000ffffffff<br>
>>> r9 00002aaaba8f9e40, r10 0000000000000008, r11 0000000000000206<br>
>>> r12 00002aaabd546cc0, r13 00000000a4400002, r14 00002aaab9a9eab0<br>
>>> r15 0000000000000009<br>
>>> /lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
>>> /lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
>>> /lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
>>> /opt/cray/lib64/libmpich_pgi.so.2(+0xcb75e) [0x2aaab96b175e]<br>
>>> ======= Backtrace: =========<br>
>>> PETSC_VERSION_RELEASE 0<br>
>>> PETSC_VERSION_MAJOR 3<br>
>>> PETSC_VERSION_MINOR 4<br>
>>> PETSC_VERSION_SUBMINOR 4<br>
>>> PETSC_VERSION_PATCH 0<br>
>>> PETSC_VERSION_DATE unknown<br>
>>> petsc_version_lt(3,3,0) is false<br>
>>> nthreads = 16 NCASES = 100<br>
>>> Application 4896333 exit codes: 127<br>
>>> Application 4896333 resources: utime ~0s, stime ~0s, Rss ~12544, inblocks ~11435, outblocks ~35849<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> /lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
>>> /lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
>>> /lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
>>> /lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
>>> /lib64/libc.so.6(__libc_malloc+0xda) [0x2aaaba60464a]<br>
>>> ./tpetsc_madams() [0x4b76d3]<br>
>>> ./tpetsc_madams() [0x57a915]<br>
>>> ./tpetsc_madams() [0x57a48e]<br>
>>> ./tpetsc_madams() [0x4a700e]<br>
>>> ./tpetsc_madams() [0x44e9be]<br>
>>> ./tpetsc_madams() [0x44e1f4]<br>
>>> /lib64/libc.so.6(__libc_start_main+0xe6) [0x2aaaba5a8c36]<br>
>>> ./tpetsc_madams() [0x44e0e9]<br>
>>> Application 4896342 exit codes: 127<br>
>>> Application 4896342 resources: utime ~0s, stime ~0s, Rss ~12544, inblocks ~11435, outblocks ~35851<br>
>>><br>
>>><br>
>>> env003> addr2line --exe=tpetsc_madams 0x4b76d3 0x57a915 0x57a48e 0x4a700e 0x44e9be 0x44e1f4<br>
>>> /autofs/na3_home1/adams/petsc/src/sys/memory/mal.c:27<br>
>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3682<br>
>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3578<br>
>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/ftn-custom/zaijf.c:14<br>
>>> /autofs/na3_home1/efdazedo/test/PETSC/./tpetsc_madams.F90:143<br>
>>> ??:0<br>
>>><br>
>>><br>
>>><br>
>>> On 04/16/2014 02:38 PM, Mark Adams wrote:<br>
>>>> Ed, I fixed some integer -- PetscInt error and put #ifdefs for the new interface to KSPSetOperators. It works for me on Edison. This is with Intel compilers.<br>
>>>> You seem to be dying in MatCreate... I'm guessing you are using 64 bit indices. I fixed this in the code. Give this a try and see if it works.<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Wed, Apr 16, 2014 at 2:33 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
>>>><br>
>>>> Mark,<br>
>>>><br>
>>>> Please send configure.log and make.log and run with 4 threads and send all output.<br>
>>>><br>
>>>> Now Ed and I have had no problem running this code. But there are some issues with running code with each thread creating their own objects. That is, I have an another example in C that does not work. There are places where we work with MPI attributes and they are not properly protected with locks. This may or may not be affecting you. If you have to develop code that has different threads create different objects you are welcome to work with Jed and I etc in getting the thread stuff working in PETSc but this branch is not the starting point. So basically Ed got lucky and we won’t have “real” support for this usage of threads for a while (months at least).<br>
>>>><br>
>>>> You absolutely should configure with —with-debugging —with-log=0<br>
>>>><br>
>>>> Barry<br>
>>>><br>
>>>><br>
>>>><br>
>>>> On Apr 16, 2014, at 10:26 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
>>>><br>
>>>>> I could also use your compile line. I am getting no output.<br>
>>>>><br>
>>>>><br>
>>>>> On Wed, Apr 16, 2014 at 10:41 AM, Ed D'Azevedo <<a href="mailto:dazevedoef@ornl.gov">dazevedoef@ornl.gov</a>> wrote:<br>
>>>>> Hi Mark,<br>
>>>>><br>
>>>>> I hope this back trace might be helpful.<br>
>>>>><br>
>>>>> You may need to build petsc with<br>
>>>>> ./configure \<br>
>>>>> --with-x=0 \<br>
>>>>> --with-debugging=0 \<br>
>>>>> --with-log=0 \<br>
>>>>><br>
>>>>><br>
>>>>> env003> addr2line --exe=tpetsc_madams 0x4b76e3 0x4aa3d7 0x484888 0x475798 0x548aa5 0x5e5df2 0x57a353 0x4a701e 0x44e9be 0x44e1f4<br>
>>>>> /autofs/na3_home1/adams/petsc/src/sys/memory/mal.c:27<br>
>>>>> /autofs/na3_home1/adams/petsc/src/sys/utils/str.c:188<br>
>>>>> /autofs/na3_home1/adams/petsc/src/sys/logging/utils/eventlog.c:317<br>
>>>>> /autofs/na3_home1/adams/petsc/src/sys/logging/plog.c:747<br>
>>>>> /autofs/na3_home1/adams/petsc/src/mat/interface/dlregismat.c:145<br>
>>>>> /autofs/na3_home1/adams/petsc/src/mat/utils/gcreate.c:57<br>
>>>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3576<br>
>>>>> /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/ftn-custom/zaijf.c:14<br>
>>>>> /autofs/na3_home1/efdazedo/test/PETSC/./tpetsc.F90:147<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> ======= Backtrace: =========<br>
>>>>> /lib64/libc.so.6(+0x75558)[0x2aaaba5ff558]<br>
>>>>> /lib64/libc.so.6(cfree+0x6c)[0x2aaaba6044fc]<br>
>>>>> ./tpetsc_madams[0x4b7759]<br>
>>>>> /lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
>>>>> /lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
>>>>> /lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
>>>>> /lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
>>>>> /lib64/libc.so.6(+0x789f0) [0x2aaaba6029f0]<br>
>>>>> /lib64/libc.so.6(__libc_malloc+0x77) [0x2aaaba6045e7]<br>
>>>>> ./tpetsc_madams() [0x4b76e3]<br>
>>>>> ./tpetsc_madams() [0x4aa3d7]<br>
>>>>> ./tpetsc_madams() [0x484888]<br>
>>>>> ./tpetsc_madams() [0x475798]<br>
>>>>> ./tpetsc_madams() [0x548aa5]<br>
>>>>> ./tpetsc_madams() [0x5e5df2]<br>
>>>>> ./tpetsc_madams() [0x57a353]<br>
>>>>> ./tpetsc_madams() [0x4a701e]<br>
>>>>> ./tpetsc_madams() [0x44e9be]<br>
>>>>> ./tpetsc_madams() [0x44e1f4]<br>
>>>>> /lib64/libc.so.6(__libc_start_main+0xe6) [0x2aaaba5a8c36]<br>
>>>>> ./tpetsc_madams() [0x44e0e9]<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On 04/16/2014 10:33 AM, Mark Adams wrote:<br>
>>>>>> cc'ing petsc-dev.<br>
>>>>>><br>
>>>>>> I will try it.<br>
>>>>>><br>
>>>>>> On Wed, Apr 16, 2014 at 10:21 AM, Ed D'Azevedo <<a href="mailto:dazevedoef@ornl.gov">dazevedoef@ornl.gov</a>> wrote:<br>
>>>>>><br>
>>>>>> Hi Mark,<br>
>>>>>><br>
>>>>>> I got an error when I tried the simple test code (see attached) on Titan.<br>
>>>>>><br>
>>>>>> Can you try to run the attached test case to see if it will work for you?<br>
>>>>>><br>
>>>>>> I have also sent the simple test code to Barry.<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> The code seems to work with 1 thread<br>
>>>>>><br>
>>>>>> env003> export OMP_NUM_THREADS=1<br>
>>>>>> env003> aprun -n 1 -d 16 ./tpetsc_madams<br>
>>>>>> PETSC_VERSION_RELEASE 0<br>
>>>>>> PETSC_VERSION_MAJOR 3<br>
>>>>>> PETSC_VERSION_MINOR 4<br>
>>>>>> PETSC_VERSION_SUBMINOR 4<br>
>>>>>> PETSC_VERSION_PATCH 0<br>
>>>>>> PETSC_VERSION_DATE unknown<br>
>>>>>> petsc_version_lt(3,3,0) is false<br>
>>>>>> nthreads = 1 NCASES = 100<br>
>>>>>> nz = 88804<br>
>>>>>> Warning: ieee_inexact is signaling<br>
>>>>>> all done<br>
>>>>>> total time is 7.268438<br>
>>>>>> maxval(err) 6.9650285539069046E-011<br>
>>>>>> Application 4895013 resources: utime ~8s, stime ~0s, Rss ~22300, inblocks ~11428, outblocks ~35848<br>
>>>>>><br>
>>>>>><br>
>>>>>> The code seems to have trouble using more threads<br>
>>>>>><br>
>>>>>> env003> export OMP_NUM_THREADS=16<br>
>>>>>> env003> aprun -n 1 -d 16 ./tpetsc_madams<br>
>>>>>> *** glibc detected *** ./tpetsc_madams: double free or corruption (!prev): 0x00000000015ee0b0 ***<br>
>>>>>> PETSC_VERSION_RELEASE 0<br>
>>>>>> PETSC_VERSION_MAJOR 3<br>
>>>>>> PETSC_VERSION_MINOR 4<br>
>>>>>> PETSC_VERSION_SUBMINOR 4<br>
>>>>>> PETSC_VERSION_PATCH 0<br>
>>>>>> PETSC_VERSION_DATE unknown<br>
>>>>>> petsc_version_lt(3,3,0) is false<br>
>>>>>> nthreads = 16 NCASES = 100<br>
>>>>>> tpetsc_madams: malloc.c:3091: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.<br>
>>>>>> Error: abort<br>
>>>>>> rax 0000000000000000, rbx 0000000000000fff, rcx ffffffffffffffff<br>
>>>>>> rdx 0000000000000006, rsp 00002aaac4554f78, rbp 00002aaad0000098<br>
>>>>>> rsi 0000000000005200, rdi 00000000000051f1, r8 00000000ffffffff<br>
>>>>>> r9 00002aaaba8f9e40, r10 0000000000000008, r11 0000000000000202<br>
>>>>>> r12 0000000000000000, r13 00002aaad0008e30, r14 0000000000000020<br>
>>>>>> r15 0000000000000000<br>
>>>>>> ======= Backtrace: =========<br>
>>>>>> /lib64/libc.so.6(+0x75558)[0x2aaaba5ff558]<br>
>>>>>> /lib64/libc.so.6(cfree+0x6c)[0x2aaaba6044fc]<br>
>>>>>> ./tpetsc_madams[0x4b7759]<br>
>>>>>> /lib64/libpthread.so.0(+0xf7c0) [0x2aaaaacdb7c0]<br>
>>>>>> /lib64/libc.so.6(gsignal+0x35) [0x2aaaba5bcb55]<br>
>>>>>> /lib64/libc.so.6(abort+0x181) [0x2aaaba5be131]<br>
>>>>>> /lib64/libc.so.6(+0x7576d) [0x2aaaba5ff76d]<br>
>>>>>> /lib64/libc.so.6(+0x789f0) [0x2aaaba6029f0]<br>
>>>>>> /lib64/libc.so.6(__libc_malloc+0x77) [0x2aaaba6045e7]<br>
>>>>>> ./tpetsc_madams() [0x4b76e3]<br>
>>>>>> ./tpetsc_madams() [0x4aa3d7]<br>
>>>>>> ./tpetsc_madams() [0x484888]<br>
>>>>>> ./tpetsc_madams() [0x475798]<br>
>>>>>> ./tpetsc_madams() [0x548aa5]<br>
>>>>>> ./tpetsc_madams() [0x5e5df2]<br>
>>>>>> ./tpetsc_madams() [0x57a353]<br>
>>>>>> ./tpetsc_madams() [0x4a701e]<br>
>>>>>> ./tpetsc_madams() [0x44e9be]<br>
>>>>>> ./tpetsc_madams() [0x44e1f4]<br>
>>>>>> /lib64/libc.so.6(__libc_start_main+0xe6) [0x2aaaba5a8c36]<br>
>>>>>> ./tpetsc_madams() [0x44e0e9]<br>
>>>>>> Application 4895021 exit codes: 127<br>
>>>>>> Application 4895021 resources: utime ~0s, stime ~0s, Rss ~12544, inblocks ~11429, outblocks ~35849<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> On 04/11/2014 04:34 PM, Mark Adams wrote:<br>
>>>>>>> PETSc dev 'master' now has Barry's thread safe stuff so you should be able to use that. I have build it in:<br>
>>>>>>><br>
>>>>>>> PETSC_DIR=/autofs/na3_home1/adams/petsc<br>
>>>>>>> PETSC_ARCH=arch-titan-opt<br>
>>>>>>><br>
>>>>>>> So try this version out. And revert the code to the repo version by doing:<br>
>>>>>>><br>
>>>>>>>> git checkout poisson.F90<br>
>>>>>>> and any other place where #if PETSC_VERSION_GE(3,5,0) is used. I only see poisson.F90.<br>
>>>>>>><br>
>>>>>>> If this works I can install it wherever you like as Ed did.<br>
>>>>>>><br>
>>>>>>> Mark<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Thu, Apr 10, 2014 at 11:24 PM, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>> wrote:<br>
>>>>>>> Hi Mark,<br>
>>>>>>><br>
>>>>>>> I think it is not a big problem until we use petsc 3.5 other than you. Now I disabled #if #endif for other users. Sorry for the inconvenience. Could you uncomment it when you are working with 3.5, and does not commit to core_dev, please?<br>
>>>>>>> We need Ed's thread safe petsc for performance issue, so we can find some way to resolve it when 3.5 is released and installed on titan.<br>
>>>>>>><br>
>>>>>>> Thanks,<br>
>>>>>>> Seung-Hoe<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Thu, Apr 10, 2014 at 11:38 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
>>>>>>> We've figured out the problem (again). It will be fixed in future versions.<br>
>>>>>>><br>
>>>>>>> We can fix your installation. I'm guessing Ed did this installation so it might not be worth fixing this code since you are up and running. Let me know what you want to do.<br>
>>>>>>><br>
>>>>>>> The problem is that is a development version of the code, which is not as stable as the releases. This version needs to be updated to get fix this problem. I have to tell you how to do this update so let me know if you want to do it.<br>
>>>>>>><br>
>>>>>>> Mark<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Thu, Apr 10, 2014 at 8:58 AM, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>> wrote:<br>
>>>>>>> One question is..<br>
>>>>>>> If PETSC_VERSION_GE is defined with PETSC_VERSION_GT and PETSC_VERSION_GT is defined with something in petsc.h,<br>
>>>>>>> redefinition of PETSC_VERSION_GT after petsc.h will change PETSC_VERSION_GE?<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Thu, Apr 10, 2014 at 10:55 AM, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>> wrote:<br>
>>>>>>> This is the code I tried to compile.<br>
>>>>>>><br>
>>>>>>> #if PETSC_VERSION_GE(5,5,0)<br>
>>>>>>> 214 BBBBBBBBBBBBBBBBBBB=0<br>
>>>>>>> 215 call KSPSetOperators(solver%ksp, solver%Amat, solver%Amat, ierr )<br>
>>>>>>> 216 #else<br>
>>>>>>> 217 AAAAAAAAAAAA=0<br>
>>>>>>> 218 call KSPSetOperators(solver%ksp, solver%Amat, solver%Amat, SAME_NONZERO_PATT ERN, ierr )<br>
>>>>>>> 219 #endif<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Thu, Apr 10, 2014 at 10:49 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Thu, Apr 10, 2014 at 8:25 AM, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>> wrote:<br>
>>>>>>> I have the same problem. I used poisson.F90 in /lustre/atlas2/env003/scratch/shku/XGC1_3_petsc_problem/<br>
>>>>>>><br>
>>>>>>> It seems that #undef should be after #include<finclude/petsc.h>. Otherwise, petsc.h seems to try redefine it.<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> Oh yes.<br>
>>>>>>><br>
>>>>>>> Anyway, I got the same error message:<br>
>>>>>>><br>
>>>>>>> PGF90-S-0038-Symbol, bbbbbbbbbbbbbbbbbbb, has not been explicitly declared (poisson.F90)<br>
>>>>>>> 0 inform, 0 warnings, 1 severes, 0 fatal for init_1field_solver<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> is this in the source file?<br>
>>>>>>><br>
>>>>>>> I tried 5.5.0 for the arguments of PETSC_VERSION_GE, but still have the same problem.<br>
>>>>>>><br>
>>>>>>> Or do you mean VERSION_GE and VERSION_LT instead of VERSION_GT and VERSION_LE?<br>
>>>>>>><br>
>>>>>>> Thanks,<br>
>>>>>>> Seung-Hoe<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Wed, Apr 9, 2014 at 9:54 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
>>>>>>> We might have a fix. It turns out that some fortran compilers to not do the #define quite right. Try this:<br>
>>>>>>><br>
>>>>>>>> git checkout poisson.F90<br>
>>>>>>> This will out the original (bad) file back. Then add to poisson.F90<br>
>>>>>>><br>
>>>>>>> #undef PETSC_VERSION_GT<br>
>>>>>>> #define PETSC_VERSION_GT(MAJOR,MINOR,SUBMINOR) \<br>
>>>>>>> (0==PETSC_VERSION_LE(MAJOR,MINOR,SUBMINOR))<br>
>>>>>>><br>
>>>>>>> This is the fix that Jed thinks will work. If it works we can propagate it.<br>
>>>>>>><br>
>>>>>>> Sorry about the confusion,<br>
>>>>>>> Mark<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Wed, Apr 9, 2014 at 7:36 PM, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>> wrote:<br>
>>>>>>> Great! Thank you.<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Wed, Apr 9, 2014 at 9:35 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
>>>>>>> Good news. I'm at a meeting and Jed Brown is here and he seems to have been able to reproduce the error. Preprocessors can be a pain. I'm going to wait until Jed has a chance to look at this and come up with a solution.<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Wed, Apr 9, 2014 at 7:28 PM, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>> wrote:<br>
>>>>>>> Now I am using iterative solver which will be replaced to your 2 field solver later.<br>
>>>>>>> Sorry. The line number 3003 is wrong. It is 209 or near it.<br>
>>>>>>> Yes. We can take a look at this when you come. It is not urgent problem.<br>
>>>>>>><br>
>>>>>>> Thanks,<br>
>>>>>>> Seung-Hoe<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Wed, Apr 9, 2014 at 9:25 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br>
>>>>>>> I noticed that your makefile was not using the new petsc solver. Is that intensional?<br>
>>>>>>><br>
>>>>>>> The line you gave me 3003 poission.F90 seems to be the last line of the file.<br>
>>>>>>><br>
>>>>>>> I am not getting Edison or Titan to build. We can take a look at this next week.<br>
>>>>>>><br>
>>>>>>> Mark<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Tue, Apr 8, 2014 at 12:38 PM, Seung-Hoe Ku <<a href="mailto:sku@pppl.gov">sku@pppl.gov</a>> wrote:<br>
>>>>>>> Hi Mark,<br>
>>>>>>><br>
>>>>>>> It seems that PETSC_VERSION_GE is not working line 3003 of poisson.F90. Is there any reason of using PETSC_VERSION_GE instead of PETSC_VERSION_LE? The previous code worked, I think.<br>
>>>>>>><br>
>>>>>>> Thanks,<br>
>>>>>>> Seung-Hoe<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>><br>
>>>>><br>
>>>><br>
><br>
<br>
</div></div></blockquote></div><br></div>