[petsc-dev] error configuring on KNL

[Satish Balay]

On Fri, 30 Jun 2017, Richard Tran Mills wrote:

> Hmm. I get the same error when adding "--download-hypre=0" to an otherwise
> functional configure script for Cori KNL. Hypre's configure doesn't like
> something about cross-compiling for KNL. It might be possible to make this
> work by building on the compute nodes, though this is, unfortunately, not
> officially supported on Cori -- NERSC staff say that builds are not
> guaranteed to work on the compute nodes, though I've managed to make it
> work sometimes.

Here is (a part of a prior email) regarding installing hypre on theta (using compute nodes).

Satish

--------------------------------

The alternative is to run configure on the KNL nodes. But this is very
slow. We could split up the build into 2 parts - only build some
packages on KNL node and the rest on front-end.

Ok - I tried "--download-hypre=1 --prefix=/home/balay/soft/hypre-xsdk-0.2.0-rc2" on the KNL node.
[this way - I can --with-hypre-dir=/home/balay/soft/hypre-xsdk-0.2.0-rc2 for subsequent builds]

This crashed halfway with 'Intel License Server' errors. The crash
happened after hypre configure completed - so I could complete the
build on the frontend.

Anyhow - here is the process"

caveat: PETSC_ARCH/externalpackages should be populated before doing
this - as compute nodes don't have external network access.

1. schedule configure run in the batch queue:

balay at thetalogin5:~/petsc.test> cat reconfigure-arch-theta-opt-cray.py
#!/usr/bin/python
if __name__ == '__main__':
  import sys
  import os
  sys.path.insert(0, os.path.abspath('config'))
  import configure
  configure_options = [
    '--download-hypre=1',
    '--prefix=/home/balay/soft/hypre-xsdk-0.2.0-rc2',
    '--with-cc=cc',
    '--with-clib-autodetect=0',
    '--with-cxx=CC',
    '--with-cxxlib-autodetect=0',
    '--with-debugging=0',
    '--with-fc=ftn',
    '--with-fortranlib-autodetect=0',
    '--with-mpiexec=mpirun',
    '--with-memalign=64',
    'COPTFLAGS=-g -O3 -fp-model fast -xMIC-AVX512',
    'CXXOPTFLAGS=-g -O3 -fp-model fast -xMIC-AVX512',
    'FOPTFLAGS=-g -O3 -fp-model fast -xMIC-AVX512',
    'LIBS=-lstdc++',
    'PETSC_ARCH=arch-theta-opt-cray',
  ]
  configure.petsc_configure(configure_options)

balay at thetalogin5:~/petsc.test> cat x.sh
#!/bin/sh
aprun -n 1 python reconfigure-arch-theta-opt-cray.py

balay at thetalogin5:~/petsc.test> qsub -t 45 -n 1 -A EarlyPerf_theta x.sh

<<<<<<<<<<<<<<<<<<

2. wait for job to complete - [or atleast hypre configure to complete].

3. if incomplete - complete the build on frontend - and install hypre libraries.

4. subsequently install PETSc using --with-hypre-dir=/home/balay/soft/hypre-xsdk-0.2.0-rc2
[on the frontend]

Satish