<div dir="ltr">OK nuked it and started over and it fixed this error below.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 8, 2014 at 7:30 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="">On Wed, May 7, 2014 at 1:02 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
</div><div class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Mark,<br>
<br>
Are you sure the PGI version is built with the correct branch of barry/make-petscoptionsobject-nonglobal ?<br></blockquote><div><br></div></div><div><div>examples/tutorials> make ex1</div><div>cc -o ex1.o -c -mp -I/autofs/na3_home1/adams/petsc/include -I/autofs/na3_home1/adams/petsc/arch-titan-opt-pgi/include -I/opt/nvidia/cudatoolkit/5.5.20-1.0402.7700.8.1/include -I/opt/nvidia/cudatoolkit/5.5.20-1.0402.7700.8.1/extras/CUPTI/include -I/opt/nvidia/cudatoolkit/5.5.20-1.0402.7700.8.1/extras/Debugger/include -I/opt/cray/udreg/2.3.2-1.0402.7546.1.5.gem/include -I/opt/cray/ugni/5.0-1.0402.7551.1.10.gem/include -I/opt/cray/pmi/5.0.1-1.0000.9799.94.9.gem/include -I/opt/cray/dmapp/4.0.1-1.0402.7784.4.1.gem/include -I/opt/cray/gni-headers/2.1-1.0402.7541.1.5.gem/include -I/opt/cray/xpmem/0.1-2.0402.45248.1.5.gem/include -I/opt/cray/rca/1.0.0-2.0402.47290.7.1.gem/include -I/opt/cray-hss-devel/7.1.0/include -I/opt/cray/krca/1.0.0-2.0402.46083.4.47.gem/include -I/opt/cray/mpt/6.2.0/gni/mpich2-pgi/121/include -I/opt/acml/5.3.1/pgi64_fma4/include -I/opt/cray/libsci/12.1.3/pgi/121/interlagos/include -I/usr/include/alps -I/opt/cray/hdf5-parallel/1.8.11/pgi/121/include -I/opt/cray/netcdf-hdf5parallel/4.3.0/pgi/121/include -I/opt/pgi/13.10.0/linux86-64/13.10/include -I/opt/cray/xe-sysroot/4.2.34/usr/include /ccs/home/adams/petsc/src/ksp/ksp/examples/tutorials/ex1.c</div>
<div>cc -mp -o ex1 ex1.o -L/autofs/na3_home1/adams/petsc/arch-titan-opt-pgi/lib -lpetsc -Wl,-rpath,/autofs/na3_home1/adams/petsc/arch-titan-opt-pgi/lib -lsuperlu_4.3 -Wl,-rpath,/opt/pgi/13.10.0/linux86/5.1/lib -L/opt/pgi/13.10.0/linux86/5.1/lib -llapack -lblas -lpthread -ldl </div>
<div>/autofs/na3_home1/adams/petsc/arch-titan-opt-pgi/lib/libpetsc.a(inode2.o): In function `MatCreate_SeqAIJ_Inode':</div><div>/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/inode2.c:90: undefined reference to `PetscOptionsPublishCount'</div>
<div>/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/inode2.c:90: undefined reference to `PetscOptionsPublishCount'</div><div>/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/inode2.c:90: undefined reference to `PetscOptionsBool'</div>
<div>/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/inode2.c:95: undefined reference to `PetscOptionsBool'</div><div>/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/inode2.c:99: undefined reference to `PetscOptionsInt'</div>
<div>/autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/inode2.c:100: undefined reference to `PetscOptionsPublishCount'</div><div>/usr/bin/ld: link errors found, deleting executable `ex1'</div></div><div><div>examples/tutorials> git status</div>
<div># On branch barry/make-petscoptionsobject-nonglobal</div><div># Changes not staged for commit:</div><div># (use "git add <file>..." to update what will be committed)</div><div># (use "git checkout -- <file>..." to discard changes in working directory)</div>
<div>#</div><div># modified: makefile</div><div>#</div><div># Untracked files:</div><div># (use "git add <file>..." to include in what will be committed)</div><div>#</div><div># ../../../../../conftest-arch-titan-opt-pgi</div>
<div># ../../../../../conftest.d</div><div># assert_mod.mod</div><div># ddt.output</div><div># omp_module.mod</div><div># tpetsc.F90</div><div>no changes added to commit (use "git add" and/or "git commit -a")</div>
</div><div><div class="h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
In that branch line 516 of aoptions.c has case OPTION_INT: not a PetscFree<br>
<br>
Meanwhile in master line 516 has ierr = PetscFree(PetscOptionsObject.title);CHKERRQ(ierr);<br>
<br>
<br>
Barry<br>
<div><div><br>
<br>
On May 7, 2014, at 2:47 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
<br>
> Barry, made the LU solver "thread safe" in that each thread calls a serial solve, for Ed (cc'ed).<br>
><br>
> Ed has a test code (attached) that can be run with 16 threads (aprun -n 1 -d 16) and NUM_OPENMP_THREADS=16. This code seems to work with Intel compilers but fails with PGI.<br>
><br>
> I've appended a stack trace. Any ideas?<br>
><br>
> Mark<br>
> Note, the line numbers are not quite right in tpetsc.F90 but the rest look OK.<br>
><br>
> #11 tpetsc () at /autofs/na3_home1/adams/petsc/src/ksp/ksp/examples/tutorials/tpetsc.F90:144 (at 0x0000000000422afa)<br>
> #10 matcreateseqaij_ () at /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/ftn-custom/zaijf.c:14 (at 0x0000000000474e95)<br>
> #9 MatCreateSeqAIJ () at /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:3574 (at 0x000000000053c8b1)<br>
> #8 MatSetType () at /autofs/na3_home1/adams/petsc/src/mat/interface/matreg.c:71 (at 0x000000000050b292)<br>
> #7 MatCreate_SeqAIJ () at /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/aij.c:4129 (at 0x000000000053f24a)<br>
> #6 MatCreate_SeqAIJ_Inode () at /autofs/na3_home1/adams/petsc/src/mat/impls/aij/seq/inode2.c:99 (at 0x000000000058b205)<br>
> #5 PetscOptionsEnd_Private () at /autofs/na3_home1/adams/petsc/src/sys/objects/aoptions.c:516 (at 0x000000000043fc25)<br>
> #4 PetscFreeAlign () at /autofs/na3_home1/adams/petsc/src/sys/memory/mal.c:72 (at 0x0000000000483f89)<br>
> #3 free () from /dsl/lib64/<a href="http://libc-2.11.3.so" target="_blank">libc-2.11.3.so</a> (at 0x00002aaab5bcc4fc)<br>
> #2 malloc_printerr () from /dsl/lib64/<a href="http://libc-2.11.3.so" target="_blank">libc-2.11.3.so</a> (at 0x00002aaab5bc7558)<br>
> #1 __libc_message () from /dsl/lib64/<a href="http://libc-2.11.3.so" target="_blank">libc-2.11.3.so</a> (at 0x00002aaab5bc1e2f)<br>
> #0 abort () from /dsl/lib64/<a href="http://libc-2.11.3.so" target="_blank">libc-2.11.3.so</a> (at 0x00002aaab5b85fb0)<br>
><br>
><br>
><br>
</div></div>> <tpetsc.F90><br>
<br>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>