<div dir="ltr">Thanks Rich, I may take you up on your offer but I've handed it over to the apps. It seems to be working, but I would not be surprised if there are more problems.<div><br></div><div>Note, here is something that I heard from ORNL about cray-petsc:</div><div><br></div><div><div><i>One of the OLCF consultants sent me this table about testing petsc ex1.f90 using pgi and petsc.</i></div><div><i><br></i></div><div><i>It shows the combination</i></div><div><i><br></i></div><div><i>pgi 17.5, cray-petsc 3.7.6.2, cudatoolkit 7.5 worked</i></div><div><i><br></i></div><div><i>At least they are aware there is a problem.</i></div></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jun 29, 2018 at 8:30 PM Richard Tran Mills <<a href="mailto:rtmills@anl.gov">rtmills@anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Mark,</div><div><br></div><div>I've got an OLCF account but don't have access to Titan. Would it be helpful for me to apply to get on whatever allocation you are using so I can poke around at this stuff? (If so, send me a message off list with details of what I should apply for.)<br></div><div><br></div><div>Hopefully I could get on there faster than it took them to enable me on Summit. I had volunteered to help with issues on Summit some time ago, but it took so long for them to enable my account on there that I didn't matter by then.</div><div><br></div><div>--Richard<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 29, 2018 at 7:12 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">Thanks all, I knew this but its been a while since I've run on this file setup.</div><div><div class="m_-5197481506513049039h5">
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Jun 29, 2018 at 9:15 AM Seung-Hoe Ku <<a href="mailto:sku@pppl.gov" target="_blank">sku@pppl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="auto">Try copying files to scratch file system. </div>
</div>
<div dir="auto">Home is not available when executive batch job. Probably interactive she'll too. </div>
<div><br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Jun 29, 2018 at 7:47 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I can not even run a job here. This is from an interactive shell:
<div><br>
</div>
<div>
<div>6<font face="monospace, monospace">4-pgi ex19-batch1:~/petsc/src/snes/examples/tutorials> make PETSC_DIR=/autofs/nccs-svm1_home1/adams/petsc PETSC_ARCH=arch-titan-dbg6</font></div>
<div><font face="monospace, monospace">cc -o ex19.o -c -fast -mp -I/autofs/nccs-svm1_home1/adams/petsc/include -I/autofs/nccs-svm1_home1/adams/petsc/arch-titan-dbg64-pgi/include -I/ccs/proj/env003/petscv3.9-dbg64-pgi/include `pwd`/ex19.c</font></div>
<div><font face="monospace, monospace">cc -fast -mp -o ex19 ex19.o -L/autofs/nccs-svm1_home1/adams/petsc/arch-titan-dbg64-pgi/lib -Wl,-rpath,/ccs/proj/env003/petscv3.9-dbg64-pgi/lib -L/ccs/proj/env003/petscv3.9-dbg64-pgi/lib -lpetsc -lHYPRE -lflapack -lfblas
-lparmetis -lmetis -lstdc++ -ldl</font></div>
<div><font face="monospace, monospace">/autofs/nccs-svm1_home1/adams/petsc/arch-titan-dbg64-pgi/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen':</font></div>
<div><font face="monospace, monospace">/autofs/nccs-svm1_home1/adams/petsc/src/sys/dll/dlimpl.c:54: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking</font></div>
<div><font face="monospace, monospace">/autofs/nccs-svm1_home1/adams/petsc/arch-titan-dbg64-pgi/lib/libpetsc.a(send.o): In function `PetscOpenSocket':</font></div>
<div><font face="monospace, monospace">/autofs/nccs-svm1_home1/adams/petsc/src/sys/classes/viewer/impls/socket/send.c:106: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used
for linking</font></div>
<div><font face="monospace, monospace">adams@titan-batch1:~/petsc/src/snes/examples/tutorials> aprun -n 1 ./ex19</font></div>
<div><font face="monospace, monospace">[NID 02258] 2018-06-29 07:45:09 Exec ./ex19 failed: chdir /autofs/nccs-svm1_home1/adams/petsc/src/snes/examples/tutorials No such file or directory</font></div>
<div><font face="monospace, monospace">adams@titan-batch1:~/petsc/src/snes/examples/tutorials> </font></div>
</div>
<div><br>
</div>
<div>Seung-Hoe runs here, maybe he can give me advice on what is wrong here.</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Jun 28, 2018 at 7:53 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
1) We need to get this PGI compiler into our regular testing. It doesn't like some Fortran code the other compilers are ok with<br>
<br>
2) Mark, likely you just need to submit the examples manually to run them. cd src/snes/examples/tutorials; make ex19 then use the batch system to submit the example
<br>
<br>
Barry<br>
<br>
<br>
> On Jun 28, 2018, at 5:35 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> I am trying to run on Titan and I am having problems running the tests. I get this output. It says it can not find a directory, but I do an ls in after the error and you can see that it is there.<br>
> <br>
> I've attached the logs.<br>
> <br>
> Any ideas?<br>
> Thanks,<br>
> Mark<br>
> <br>
> adams@titan-login6:~/petsc> make PETSC_DIR=/ccs/proj/env003/petscv3.9-opt64-pgi PETSC_ARCH="" test<br>
> Running test examples to verify correct installation<br>
> Using PETSC_DIR=/ccs/proj/env003/petscv3.9-opt64-pgi and PETSC_ARCH=<br>
> *******************Error detected during compile or link!*******************<br>
> See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> /ccs/home/adams/petsc/src/snes/examples/tutorials ex19<br>
> *********************************************************************************<br>
> cc -o ex19.o -c -fast -mp -I/ccs/proj/env003/petscv3.9-opt64-pgi/include -I/ccs/proj/env003/petscv3.9-opt64-pgi/include -I/ccs/proj/env003/petscv3.9-opt64-pgi/include `pwd`/ex19.c<br>
> cc -fast -mp -o ex19 ex19.o -L/ccs/proj/env003/petscv3.9-opt64-pgi/lib -Wl,-rpath,/ccs/proj/env003/petscv3.9-opt64-pgi/lib -L/ccs/proj/env003/petscv3.9-opt64-pgi/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -lstdc++ -ldl<br>
> /ccs/proj/env003/petscv3.9-opt64-pgi/lib/libpetsc.a(dlimpl.o): In function `PetscDLOpen':<br>
> /autofs/nccs-svm1_home1/adams/petsc/src/sys/dll/dlimpl.c:53: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking<br>
> /ccs/proj/env003/petscv3.9-opt64-pgi/lib/libpetsc.a(send.o): In function `PetscOpenSocket':<br>
> /autofs/nccs-svm1_home1/adams/petsc/src/sys/classes/viewer/impls/socket/send.c:101: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking<br>
> /bin/rm -f ex19.o<br>
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI process<br>
> See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> [NID 18525] 2018-06-28 18:25:39 Exec ./ex19 failed: chdir /autofs/nccs-svm1_home1/adams/petsc/src/snes/examples/tutorials No such file or directory<br>
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI processes<br>
> See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> [NID 18525] 2018-06-28 18:25:46 Exec ./ex19 failed: chdir /autofs/nccs-svm1_home1/adams/petsc/src/snes/examples/tutorials No such file or directory<br>
> 1,5c1<br>
> < lid velocity = 0.0016, prandtl # = 1., grashof # = 1.<br>
> < 0 SNES Function norm 0.0406612 <br>
> < 1 SNES Function norm 4.12227e-06 <br>
> < 2 SNES Function norm 6.098e-11 <br>
> < Number of SNES iterations = 2<br>
> ---<br>
> > [NID 18525] 2018-06-28 18:25:55 Exec ./ex19 failed: chdir /autofs/nccs-svm1_home1/adams/petsc/src/snes/examples/tutorials No such file or directory<br>
> /ccs/home/adams/petsc/src/snes/examples/tutorials<br>
> Possible problem with ex19_hypre, diffs above<br>
> =========================================<br>
> *******************Error detected during compile or link!*******************<br>
> See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/documentation/faq.html</a><br>
> /ccs/home/adams/petsc/src/snes/examples/tutorials ex5f<br>
> *********************************************************<br>
> ftn -c -fast -mp -I/ccs/proj/env003/petscv3.9-opt64-pgi/include -I/ccs/proj/env003/petscv3.9-opt64-pgi/include -I/ccs/proj/env003/petscv3.9-opt64-pgi/include -o ex5f.o ex5f.F90<br>
> PGF90-S-0155-Could not resolve generic procedure dmdacreate2d (ex5f.F90: 120)<br>
> PGF90-S-0450-Argument number 2 to snesgetiterationnumber: kind mismatch (ex5f.F90: 180)<br>
> 0 inform, 0 warnings, 2 severes, 0 fatal for main<br>
> PGF90-S-0155-Could not resolve generic procedure matsetvalueslocal (ex5f.F90: 464)<br>
> PGF90-S-0155-Could not resolve generic procedure matsetvalueslocal (ex5f.F90: 477)<br>
> 0 inform, 0 warnings, 2 severes, 0 fatal for formjacobianlocal<br>
> gmake[3]: [ex5f.o] Error 2 (ignored)<br>
> ftn -fast -mp -o ex5f ex5f.o -L/ccs/proj/env003/petscv3.9-opt64-pgi/lib -Wl,-rpath,/ccs/proj/env003/petscv3.9-opt64-pgi/lib -L/ccs/proj/env003/petscv3.9-opt64-pgi/lib -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -lstdc++ -ldl<br>
> /usr/bin/ld: cannot find ex5f.o: No such file or directory<br>
> /usr/bin/sha1sum: ex5f: No such file or directory<br>
> gmake[3]: [ex5f] Error 2 (ignored)<br>
> /bin/rm -f ex5f.o<br>
> Completed test examples<br>
> adams@titan-login6:~/petsc> ll /autofs/nccs-svm1_home1/adams/petsc/src/snes/examples/tutorials<br>
> total 1864<br>
> -rw-rw-r-- 1 adams adams 11134 May 3 14:26 ex1.c<br>
> drwxr-xr-x 3 adams adams 4096 May 3 14:26 ex10d<br>
> -rw-rw-r-- 1 adams adams 92615 May 3 14:26 ex12.c<br>
> -rw-r--r-- 1 adams adams 19324 Jun 21 17:59 ex13.c<br>
> -rw-rw-r-- 1 adams adams 20550 May 3 14:26 ex14.c<br>
> <br>
> <make.log><configure.log><br>
<br>
</blockquote>
</div>
</blockquote>
</div>
</div>
-- <br>
<div dir="ltr" class="m_-5197481506513049039m_8106405693304712121m_788250796498798001gmail_signature" data-smartmail="gmail_signature">
Sent from Iphone</div>
</blockquote>
</div>
</div></div></div>
</blockquote></div><br></div>
</blockquote></div>