<div dir="ltr"><div>Mark, were you trying this in Valgrind with a binary targeting KNL, i.e., built to use AVX-512 instructions? I don't think Valgrind implements all (or any?) of those, so a failure is not a surprise. Indeed, I've had Valgrind choke on some AVX2 instructions, though maybe the most recent versions of Valgrind will handle these properly now.<br></div><div><br></div><div>--Richard<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 3, 2018 at 4:25 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">Well this does work without valgrind.</div><div><div class="h5">
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Jul 3, 2018 at 6:36 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I built a 32 bit integer version and now it dies in PetscInit. Ugh.
<div><br>
</div>
<div>
<div>==3965== Conditional jump or move depends on uninitialised value(s)</div>
<div>==3965== at 0x27AFFD8F: _int_free (malloc.c:3945)</div>
<div>==3965== by 0x20074F48: PetscOptionsSetValue (options.c:1152)</div>
<div>==3965== by 0x20070450: PetscOptionsInsertArgs_Private (options.c:636)</div>
<div>==3965== by 0x2007189F: PetscOptionsInsert (options.c:746)</div>
<div>==3965== by 0x20093DB1: PetscInitialize (pinit.c:929)</div>
<div>==3965== by 0x2000ABCE: main (ex19.c:106)</div>
<div>==3965== </div>
<div>vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0x7F 0x8 0x7B 0xC2 0xC5 0xFB 0x10 0xD</div>
<div>vex amd64->IR: REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0</div>
<div>vex amd64->IR: VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE</div>
<div>vex amd64->IR: PFX.66=0 PFX.F2=0 PFX.F3=0</div>
<div>==3965== valgrind: Unrecognised instruction at address 0x20104a53.</div>
<div>==3965== at 0x20104A53: kh_resize_HTPrinted (viewreg.c:12)</div>
<div>==3965== by 0x20104EE2: kh_put_HTPrinted (viewreg.c:12)</div>
<div>==3965== by 0x201061EF: PetscOptionsHelpPrintedCheck (viewreg.c:89)</div>
<div>==3965== by 0x2009AD61: PetscOptionsBegin_Private (aoptions.c:34)</div>
<div>==3965== by 0x2008362E: PetscOptionsSetFromOptions (options.c:2646)</div>
<div>==3965== by 0x20094689: PetscInitialize (pinit.c:967)</div>
<div>==3965== by 0x2000ABCE: main (ex19.c:106)</div>
<div>==3965== Your program just tried to execute an instruction that Valgrind</div>
<div>==3965== did not recognise. There are two possible reasons for this.</div>
<div>==3965== 1. Your program has a bug and erroneously jumped to a non-code</div>
<div>==3965== location. If you are running Memcheck and you just saw a</div>
<div>==3965== warning about a bad jump, it's probably your program's fault.</div>
<div>==3965== 2. The instruction is legitimate but Valgrind doesn't handle it,</div>
<div>==3965== i.e. it's Valgrind's fault. If you think this is the case or</div>
<div>==3965== you are not sure, please let us know and we'll try to fix it.</div>
<div>==3965== Either way, Valgrind will now raise a SIGILL signal which will</div>
<div>==3965== probably kill your program.</div>
<div>[0]PETSC ERROR: ==3965== Conditional jump or move depends on uninitialised value(s)</div>
<div>==3965== at 0x27B18B48: strchrnul (strchr.S:106)</div>
<div>==3965== by 0x27AE30C8: __find_specmb (printf-parse.h:108)</div>
<div>==3965== by 0x27AE30C8: vfprintf (vfprintf.c:1311)</div>
<div>==3965== by 0x27AFA045: vsnprintf (vsnprintf.c:119)</div>
<div>==3965== by 0x20116802: PetscVSNPrintf (mprint.c:178)</div>
<div>==3965== by 0x20117006: PetscVFPrintfDefault (mprint.c:294)</div>
<div>==3965== by 0x20136BB2: PetscErrorPrintfDefault (errtrace.c:116)</div>
<div>==3965== by 0x2013938C: PetscSignalHandlerDefault (signal.c:131)</div>
<div>==3965== by 0x2013903B: PetscSignalHandler_Private (signal.c:43)</div>
<div>==3965== by 0x2335D07F: ??? (in /global/u2/m/madams/petsc_<wbr>install/petsc/src/snes/<wbr>examples/tutorials/ex19)</div>
<div>==3965== by 0x20104A52: kh_resize_HTPrinted (viewreg.c:12)</div>
<div>==3965== by 0x20104EE2: kh_put_HTPrinted (viewreg.c:12)</div>
<div>==3965== by 0x201061EF: PetscOptionsHelpPrintedCheck (viewreg.c:89)</div>
<div>==3965== </div>
<div>==3965== Conditional jump or move depends on uninitialised value(s)</div>
<div>==3965== at 0x27B08184: strlen (strlen.S:210)</div>
<div>==3965== by 0x200B3A48: PetscStrlen (str.c:158)</div>
<div>==3965== by 0x2011693D: PetscVSNPrintf (mprint.c:188)</div>
<div>==3965== by 0x20117006: PetscVFPrintfDefault (mprint.c:294)</div>
<div>==3965== by 0x20136BB2: PetscErrorPrintfDefault (errtrace.c:116)</div>
<div>==3965== by 0x2013938C: PetscSignalHandlerDefault (signal.c:131)</div>
<div>==3965== by 0x2013903B: PetscSignalHandler_Private (signal.c:43)</div>
<div>==3965== by 0x2335D07F: ??? (in /global/u2/m/madams/petsc_<wbr>install/petsc/src/snes/<wbr>examples/tutorials/ex19)</div>
<div>==3965== by 0x20104A52: kh_resize_HTPrinted (viewreg.c:12)</div>
<div>==3965== by 0x20104EE2: kh_put_HTPrinted (viewreg.c:12)</div>
<div>==3965== by 0x201061EF: PetscOptionsHelpPrintedCheck (viewreg.c:89)</div>
<div>==3965== by 0x2009AD61: PetscOptionsBegin_Private (aoptions.c:34)</div>
<div>==3965== </div>
<div>------------------------------<wbr>------------------------------<wbr>------------</div>
<div>[0]PETSC ERROR: Caught signal number 4 Illegal instruction: Likely due to memory corruption</div>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Mon, Jul 2, 2018 at 8:53 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Looping Treb and Baky back into the thread, and dropping PETSs.
<div><br>
</div>
<div>Great, thanks Barry for figuring this out.</div>
<div><br>
</div>
<div>Treb and Baky need 64 bit indices, but in the mean time I can build a 32 bit version to let them test.</div>
<div><br>
</div>
<div>I am all set up to test a 64 bit version. If you can give me a branch I can test.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Mon, Jul 2, 2018 at 7:47 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Progress has been made. These libraries do contain mkl_set_num_threads() is found so ./configure knows that it is MKL libraries (unlike before when it did not recognize that it was MKL libraries).<br>
<br>
<br>
Damn it, here is why: <br>
<br>
--with-64-bit-indices=1<br>
<br>
currently the MKL sparse stuff only works with 32 bit integers AND 32 bit integer BLAS/LAPACK.<br>
<br>
It will never work with 64 bit indices and 32 bit integer BLAS/LAPACK. but maybe could be upgraded to work with 64 bit indices and 64 bit integer BLAS/LAPACK Richard?<br>
<br>
<br>
Anyways the requirement in mkl_sparse.py is <br>
<br>
self.requires32bitint = 1<br>
<br>
The problem is ./configure does not print enough information to make it immediately clear it is rejecting the package because of this incompatibility.<br>
<br>
<br>
<br>
Barry<br>
<br>
<br>
<br>
<br>
> On Jul 2, 2018, at 5:57 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> Same error:<br>
> <br>
> 15:53 nid02517 master *= ~/petsc_install/petsc/src/<wbr>snes/examples/tutorials$ make PETSC_DIR=/global/homes/m/<wbr>madams/petsc_install/petsc-<wbr>cori-knl-dbg64-intel-omp PETSC_ARCH="" ex19<br>
> cc -o ex19.o -c -g -O0 -mkl -static-intel -fopenmp -I/global/homes/m/madams/<wbr>petsc_install/petsc-cori-knl-<wbr>dbg64-intel-omp/include -I/global/homes/m/madams/<wbr>petsc_install/petsc-cori-knl-<wbr>dbg64-intel-omp/include -I/global/homes/m/madams/tmp/<wbr>hypre-2.14.0/include
-I/opt/intel/compilers_and_<wbr>libraries_2018.1.163/linux/<wbr>mkl/include -I/global/homes/m/madams/<wbr>petsc_install/petsc-cori-knl-<wbr>dbg64-intel-omp/include `pwd`/ex19.c
<br>
> cc -g -O0 -mkl -static-intel -fopenmp -o ex19 ex19.o -L/global/homes/m/madams/<wbr>petsc_install/petsc-cori-knl-<wbr>dbg64-intel-omp/lib -Wl,-rpath,/global/homes/m/<wbr>madams/tmp/hypre-2.14.0/lib -L/global/homes/m/madams/tmp/<wbr>hypre-2.14.0/lib -Wl,-rpath,/global/homes/m/<wbr>madams/petsc_install/petsc-<wbr>cori-knl-dbg64-intel-omp/lib
-L/global/homes/m/madams/<wbr>petsc_install/petsc-cori-knl-<wbr>dbg64-intel-omp/lib -L/opt/intel/compilers_and_<wbr>libraries_2018.1.163/linux/<wbr>mkl/lib/intel64 -lpetsc -lHYPRE -lparmetis -lmetis -lstdc++ -ldl -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread
<br>
> /opt/intel/compilers_and_<wbr>libraries_2018.1.163/linux/<wbr>mkl/lib/intel64/libmkl_core.a(<wbr>mkl_semaphore.o): In function `mkl_serv_load_inspector':
<br>
> mkl_semaphore.c:(.text+0x123): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking <br>
> /global/homes/m/madams/petsc_<wbr>install/petsc-cori-knl-dbg64-<wbr>intel-omp/lib/libpetsc.a(send.<wbr>o): In function `PetscOpenSocket':
<br>
> /global/u2/m/madams/petsc_<wbr>install/petsc/src/sys/classes/<wbr>viewer/impls/socket/send.c:<wbr>108: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
<br>
> rm ex19.o <br>
> 15:54 nid02517 master *= ~/petsc_install/petsc/src/<wbr>snes/examples/tutorials$ make PETSC_DIR=/global/homes/m/<wbr>madams/petsc_install/petsc-<wbr>cori-knl-dbg64-intel-omp PETSC_ARCH="" runex19_gamg<br>
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
<br>
> [0]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>-- <br>
> [1]PETSC ERROR: --------------------- Error Message ------------------------------<wbr>------------------------------<wbr>-- <br>
> [1]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: <a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#external" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/<wbr>documentation/installation.<wbr>html#external</a>
<br>
> [1]PETSC ERROR: Unknown Mat type given: aijmkl <br>
> <br>
> On Mon, Jul 2, 2018 at 6:43 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> Hm - I suspect its an issue with mkl includes - not the libraries.<br>
> <br>
> Satish<br>
> <br>
> On Mon, 2 Jul 2018, Smith, Barry F. wrote:<br>
> <br>
> > <br>
> > Mark,<br>
> > <br>
> > This is not useful. We need the new configure log from when you list all the libraries NERSE recommends (which may work).<br>
> > <br>
> > I already said what was wrong with this configuration.<br>
> > <br>
> > Barry<br>
> > <br>
> > <br>
> > > On Jul 2, 2018, at 5:25 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> > > <br>
> > > <br>
> > > <br>
> > > On Mon, Jul 2, 2018 at 6:22 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> > > I don't understand the problem here..<br>
> > > <br>
> > > > > > [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing package:
<a href="http://www.mcs.anl.gov/petsc/documentation/installation.html#external" rel="noreferrer" target="_blank">
http://www.mcs.anl.gov/petsc/<wbr>documentation/installation.<wbr>html#external</a><br>
> > > > > > [0]PETSC ERROR: Unknown Mat type given: aijmkl<br>
> > > <br>
> > > If this is the problem - then we'll have to look at configure.log to check why PETSC_HAVE_MKL_SPARSE flag is not set.<br>
> > > <br>
> > > <configure.log><br>
> > <br>
> <br>
> <configure.log><br>
<br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div></div></div>
</blockquote></div><br></div>