<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">As requested, I attached the configure.log and make.log files<o:p></o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal">Luke<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Barry Smith <bsmith@petsc.dev><br>
<b>Date: </b>Friday, March 11, 2022 at 2:47 PM<br>
<b>To: </b>"Roskop, Luke B" <luke.roskop@hpe.com><br>
<b>Cc: </b>"petsc-users@mcs.anl.gov" <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] Problem building petsc +rocm variant<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"> Please send configure.log and make.log to
<a href="mailto:petsc-maint@mcs.anl.gov">petsc-maint@mcs.anl.gov</a> For some reason PETSc's configure does not detect that the MPI does provide MPI_Type_dup() even though it is prototyped in the MPI include file.<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in">On Mar 11, 2022, at 3:34 PM, Roskop, Luke B <<a href="mailto:luke.roskop@hpe.com">luke.roskop@hpe.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Hi, I’m hoping you can help me figure out how to build PETSc targeting AMDGPUs (gfx90a GPU).<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I’m attempting to build on the CrayEx ORNL system called crusher, using ROCmCC (AMD’s compiler), and cray-mpich. In case it helps, I’m using spack to build petsc with the “petsc@main%rocmcc+batch+rocm amdgpu_target=gfx90a”
spec.<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Spack ends up invoking the following configure for PETSc:<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">'/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/python-3.9.10-7y7mxajn5rywz5xdnba4azphcdodxiub/bin/python3.9'
'configure' '--prefix=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/petsc-main-mccbycx66la7rlx6jv44f6zd63cmdzm7' '--with-ssl=0' '--download-c2html=0' '--download-sowing=0' '--download-hwloc=0' 'CFLAGS=' 'FFLAGS=-fPIC'
'CXXFLAGS=' 'LDFLAGS=-Wl,-z,notext' '--with-cc=/opt/cray/pe/mpich/8.1.12/ofi/amd/4.4/bin/mpicc' '--with-cxx=/opt/cray/pe/mpich/8.1.12/ofi/amd/4.4/bin/mpicxx' '--with-fc=/opt/cray/pe/mpich/8.1.12/ofi/amd/4.4/bin/mpif90' '--with-precision=double' '--with-scalar-type=real'
'--with-shared-libraries=1' '--with-debugging=0' '--with-openmp=0' '--with-64-bit-indices=0' 'COPTFLAGS=' 'FOPTFLAGS=' 'CXXOPTFLAGS=' '--with-blas-lapack-lib=/opt/cray/pe/libsci/21.08.1.2/AMD/4.0/x86_64/lib/libsci_amd.so' '--with-batch=1' '--with-x=0' '--with-clanguage=C'
'--with-cuda=0' '--with-hip=1' '--with-hip-dir=/opt/rocm-4.5.0/hip' '--with-metis=1' '--with-metis-include=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/metis-5.1.0-zn5rn5srr7qzxyo5tq36d46adcsyc5a7/include'
'--with-metis-lib=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/metis-5.1.0-zn5rn5srr7qzxyo5tq36d46adcsyc5a7/lib/libmetis.so' '--with-hypre=1' '--with-hypre-include=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/hypre-develop-vgfx3lhhloq4cnethsrrpz7iez7x6wad/include'
'--with-hypre-lib=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/hypre-develop-vgfx3lhhloq4cnethsrrpz7iez7x6wad/lib/libHYPRE.so' '--with-parmetis=1' '--with-parmetis-include=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/parmetis-4.0.3-6jqxqmt7qqq73rxmx3beu5ba4vj3r253/include'
'--with-parmetis-lib=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/parmetis-4.0.3-6jqxqmt7qqq73rxmx3beu5ba4vj3r253/lib/libparmetis.so' '--with-superlu_dist=1' '--with-superlu_dist-include=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/superlu-dist-develop-mpiyhomp4k72bilqn6xk7uol36ulsdve/include'
'--with-superlu_dist-lib=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/superlu-dist-develop-mpiyhomp4k72bilqn6xk7uol36ulsdve/lib/libsuperlu_dist.so' '--with-ptscotch=0' '--with-suitesparse=0' '--with-hdf5=1'
'--with-hdf5-include=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/hdf5-1.12.1-dp5vqo4tjh6oi7szpcsqkdlifgjxknzf/include' '--with-hdf5-lib=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/hdf5-1.12.1-dp5vqo4tjh6oi7szpcsqkdlifgjxknzf/lib/libhdf5.so'
'--with-zlib=1' '--with-zlib-include=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/zlib-1.2.11-2ciasfxwyxanyohroisdpvidg4gs2fdy/include' '--with-zlib-lib=/gpfs/alpine/ven114/scratch/lukebr/confidence/spack/install_tree/cray-sles15-zen3/rocmcc-4.5.0/zlib-1.2.11-2ciasfxwyxanyohroisdpvidg4gs2fdy/lib/libz.so'
'--with-mumps=0' '--with-trilinos=0' '--with-fftw=0' '--with-valgrind=0' '--with-gmp=0' '--with-libpng=0' '--with-giflib=0' '--with-mpfr=0' '--with-netcdf=0' '--with-pnetcdf=0' '--with-moab=0' '--with-random123=0' '--with-exodusii=0' '--with-cgns=0' '--with-memkind=0'
'--with-p4est=0' '--with-saws=0' '--with-yaml=0' '--with-hwloc=0' '--with-libjpeg=0' '--with-scalapack=1' '--with-scalapack-lib=/opt/cray/pe/libsci/21.08.1.2/AMD/4.0/x86_64/lib/libsci_amd.so' '--with-strumpack=0' '--with-mmg=0' '--with-parmmg=0' '--with-tetgen=0'
'--with-cxx-dialect=C++11'</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Using spack, I see this error at compile time:<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">/tmp/lukebr/spack-stage/spack-stage-petsc-main-5jlv6jcfdaa37iy5zm77umvb6uvgwdo7/spack-src/src/vec/is/sf/impls/basic/sfpack.c:463:19: error: static
declaration of 'MPI_Type_dup' follows non-static declaration</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">static inline int MPI_Type_dup(MPI_Datatype datatype,MPI_Datatype *newtype)</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo"> </span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">^</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">/opt/cray/pe/mpich/8.1.12/ofi/cray/10.0/include/mpi.h:1291:5: note: previous declaration is here</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">int MPI_Type_dup(MPI_Datatype oldtype, MPI_Datatype *newtype) MPICH_API_PUBLIC;</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo"> </span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">^</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">1 error generated.</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">To get around this error, I pass “-DPETSC_HAVE_MPI_TYPE_DUP” but then I see the following lining error:<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">CLINKER arch-linux-c-opt/lib/libpetsc.so.3.016.5</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">ld.lld: error: undefined hidden symbol: PetscSFCreate_Window</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">>>> referenced by sfregi.c</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">>>></span></span><span class="apple-converted-space"><span style="font-size:8.5pt;font-family:Menlo"> </span></span><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">arch-linux-c-opt/obj/vec/is/sf/interface/sfregi.o:(PetscSFRegisterAll)</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">clang-13: error: linker command failed with exit code 1 (use -v to see invocation)</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">gmake[3]: *** [gmakefile:113: arch-linux-c-opt/lib/libpetsc.so.3.016.5] Error 1</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div style="margin-left:.5in">
<p class="MsoNormal" style="margin-left:.5in"><span class="s1"><span style="font-size:8.5pt;font-family:Menlo">gmake[2]: *** [/tmp/lukebr/spack-stage/spack-stage-petsc-main-5jlv6jcfdaa37iy5zm77umvb6uvgwdo7/spack-src/lib/petsc/conf/rules:56: libs] Error 2</span></span><span style="font-size:8.5pt;font-family:Menlo"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Before I continue, is there a preferred way to build PETSc on an AMDGPU system? Could you share this?<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Thanks,<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Luke<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</div>
</body>
</html>