<div dir="ltr">Yes, I submitted jobs before I went to bed last night (in the UK) and it configure and built. So far so good.<div>Thanks,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 30, 2017 at 11:32 PM, Richard Tran Mills <span dir="ltr"><<a href="mailto:rtmills@anl.gov" target="_blank">rtmills@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 30, 2017 at 3:27 PM, Balay, Satish <span dir="ltr"><<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><span>On Fri, 30 Jun 2017, Richard Tran Mills wrote:<br>
<br>
> Hmm. I get the same error when adding "--download-hypre=0" to an otherwise<br>
> functional configure script for Cori KNL. Hypre's configure doesn't like<br>
> something about cross-compiling for KNL. It might be possible to make this<br>
> work by building on the compute nodes, though this is, unfortunately, not<br>
> officially supported on Cori -- NERSC staff say that builds are not<br>
> guaranteed to work on the compute nodes, though I've managed to make it<br>
> work sometimes.<br>
<br>
</span></span><span class="">Here is (a part of a prior email) regarding installing hypre on theta (using compute nodes).<br></span></blockquote><div><br></div><div>The approach Satish outlines below sounds like the most reasonable course of action we ahve. I wonder if it might help a little bit to try to just run the standalone Hypre configuration (using the incantation in the configure.log) in the batch script, rather than invoking PETSc's configure.py.<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888"><div>--Richard<br><br></div></font></span><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Satish<br>
<br>
------------------------------<wbr>--<br>
<br>
The alternative is to run configure on the KNL nodes. But this is very<br>
slow. We could split up the build into 2 parts - only build some<br>
packages on KNL node and the rest on front-end.<br>
<br>
Ok - I tried "--download-hypre=1 --prefix=/home/balay/soft/hypr<wbr>e-xsdk-0.2.0-rc2" on the KNL node.<br>
[this way - I can --with-hypre-dir=/home/balay/s<wbr>oft/hypre-xsdk-0.2.0-rc2 for subsequent builds]<br>
<br>
This crashed halfway with 'Intel License Server' errors. The crash<br>
happened after hypre configure completed - so I could complete the<br>
build on the frontend.<br>
<br>
Anyhow - here is the process"<br>
<br>
caveat: PETSC_ARCH/externalpackages should be populated before doing<br>
this - as compute nodes don't have external network access.<br>
<br>
1. schedule configure run in the batch queue:<br>
<br>
balay@thetalogin5:~/petsc.test<wbr>> cat reconfigure-arch-theta-opt-cra<wbr>y.py<br>
#!/usr/bin/python<br>
if __name__ == '__main__':<br>
import sys<br>
import os<br>
sys.path.insert(0, os.path.abspath('config'))<br>
import configure<br>
configure_options = [<br>
'--download-hypre=1',<br>
'--prefix=/home/balay/soft/hyp<wbr>re-xsdk-0.2.0-rc2',<br>
'--with-cc=cc',<br>
'--with-clib-autodetect=0',<br>
'--with-cxx=CC',<br>
'--with-cxxlib-autodetect=0',<br>
'--with-debugging=0',<br>
'--with-fc=ftn',<br>
'--with-fortranlib-autodetect=<wbr>0',<br>
'--with-mpiexec=mpirun',<br>
'--with-memalign=64',<br>
'COPTFLAGS=-g -O3 -fp-model fast -xMIC-AVX512',<br>
'CXXOPTFLAGS=-g -O3 -fp-model fast -xMIC-AVX512',<br>
'FOPTFLAGS=-g -O3 -fp-model fast -xMIC-AVX512',<br>
'LIBS=-lstdc++',<br>
'PETSC_ARCH=arch-theta-opt-cra<wbr>y',<br>
]<br>
configure.petsc_configure(conf<wbr>igure_options)<br>
<br>
balay@thetalogin5:~/petsc.test<wbr>> cat x.sh<br>
#!/bin/sh<br>
aprun -n 1 python reconfigure-arch-theta-opt-cra<wbr>y.py<br>
<br>
balay@thetalogin5:~/petsc.test<wbr>> qsub -t 45 -n 1 -A EarlyPerf_theta x.sh<br>
<br>
<<<<<<<<<<<<<<<<<<<br>
<br>
2. wait for job to complete - [or atleast hypre configure to complete].<br>
<br>
3. if incomplete - complete the build on frontend - and install hypre libraries.<br>
<br>
4. subsequently install PETSc using --with-hypre-dir=/home/balay/s<wbr>oft/hypre-xsdk-0.2.0-rc2<br>
[on the frontend]<br>
<span class="m_3014400280238645311HOEnZb"><font color="#888888"><br>
Satish<br>
<br>
<br>
</font></span></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>