<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Arial;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New",serif;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier",serif;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0cm;
margin-bottom:.0001pt;
background:white;
font-size:8.5pt;
font-family:"Menlo",sans-serif;
color:black;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0cm;
margin-bottom:.0001pt;
background:white;
font-size:8.5pt;
font-family:"Menlo",sans-serif;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:595.0pt 842.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Phil,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sure, test_scalapack.f90 call ScaLAPACK routines and uses inside MPI library. The output of the failing ftn command with the craype-verbose flag<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> ftn -o exe test_scalapack.f90 -craype-verbose <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">ftn_driver.exe -hcpu=haswell -hstatic -D__CRAYXC -D__CRAY_HASWELL -D__CRAYXT_COMPUTE_LINUX_TARGET -hnetwork=aries -o exe test_scalapack.f90 -Wl,-rpath=/opt/cray/pe/cce/8.6.5/cce/x86_64/lib
-ulast_user_arg -I /opt/cray/pe/cce/8.6.5/cce/x86_64/include/craylibs -I/usr/include -L/opt/gcc/6.1.0/snos/lib64 -Wl,-rpath=/opt/cray/pe/gcc-libs -L/usr/lib64 -L/lib64 -ugcc_base=/opt/gcc/6.1.0/snos -uno_driver_libs -Wl,/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/libmodules.a,/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/libomp.a,/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/libopenacc.a,
-I /usr/include -I/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/include -I/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/include -I/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/include -I/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/include -I/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/include
-I/opt/cray/gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari/include -I/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/include -I/opt/cray/pe/pmi/5.0.13/include -I/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/include -I/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/include
-I/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/include -I/opt/cray/krca/2.2.2-6.0.5.0_5.46__g4614cf3.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/dmapp/default/lib64
-L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib -L/sw/xc40cle6/darshan/3.1.4/sles12.3_gcc7.2.0/lib -L/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/lib64 -L/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/lib64
-L/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/lib64 -L/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/lib64 -L/opt/cray/pe/pmi/5.0.13/lib64 -L/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/lib64 -L/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/lib64
-L/opt/cray/pe/atp/2.1.1/libApp -L/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/pkgconfig/../ -L/lib64 -L/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/lib64 -Wl,@/sw/xc40cle6/darshan/3.1.4/sles12.3_gcc7.2.0/share/ld-opts/darshan-base-ld-opts -lfmpich -lmpichcxx
-Wl,--start-group -ldarshan -ldarshan-stubs -Wl,--end-group -lz -Wl,--no-as-needed,-lAtpSigHandler,-lAtpSigHCommData -Wl,--undefined=_ATP_Data_Globals -Wl,--undefined=__atpHandlerInstall -lpthread -lmpichf90_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mpi_mp
-lm -lf -lsci_cray_mp -lmpich_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mp -lcraymp -lm -lpthread -lf -lhugetlbfs -lpgas-dmapp -lfi -lu -lrt -Wl,--undefined=dmapp_get_flag_nbi -ldmapp -lugni -ludreg -lpthread -lm -lcray-c++-rts -lstdc++ -lxpmem -ldmapp -lpthread
-lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lugni -lpthread -lalpsutil -lpthread -lrca -ludreg -lquadmath -lm -lomp -lcraymp -lpthread -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lmodules -lm -lfi -lm -lquadmath -lcraymath -lm -lgfortran -lquadmath -lf -lm
-lpthread -lu -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lcsup -Wl,--as-needed,-latomic,--no-as-needed -Wl,--as-needed,-lgfortran,--no-as-needed -lstdc++ -lpthread -Wl,--start-group,-lc,-lcsup,-lgcc_eh,-lm,-lgcc,--end-group -Wl,-T/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/2.23.1.cce.ld <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> module swap darshan/3.1.4 darshan/3.1.5 <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> ftn -o exe test_scalapack.f90 -craype-verbose <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">ftn_driver.exe -hcpu=haswell -hstatic -D__CRAYXC -D__CRAY_HASWELL -D__CRAYXT_COMPUTE_LINUX_TARGET -hnetwork=aries -o exe test_scalapack.f90 -Wl,-rpath=/opt/cray/pe/cce/8.6.5/cce/x86_64/lib
-ulast_user_arg -I /opt/cray/pe/cce/8.6.5/cce/x86_64/include/craylibs -I/usr/include -L/opt/gcc/6.1.0/snos/lib64 -Wl,-rpath=/opt/cray/pe/gcc-libs -L/usr/lib64 -L/lib64 -ugcc_base=/opt/gcc/6.1.0/snos -uno_driver_libs -Wl,/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/libmodules.a,/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/libomp.a,/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/libopenacc.a,
-I /usr/include -I/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/include -I/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/include -I/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/include -I/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/include -I/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/include
-I/opt/cray/gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari/include -I/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/include -I/opt/cray/pe/pmi/5.0.13/include -I/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/include -I/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/include
-I/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/include -I/opt/cray/krca/2.2.2-6.0.5.0_5.46__g4614cf3.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/dmapp/default/lib64
-L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib -L/sw/xc40cle6/darshan/3.1.5/sles12.3_gcc7.2.0/lib -L/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/lib64 -L/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/lib64
-L/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/lib64 -L/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/lib64 -L/opt/cray/pe/pmi/5.0.13/lib64 -L/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/lib64 -L/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/lib64
-L/opt/cray/pe/atp/2.1.1/libApp -L/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/pkgconfig/../ -L/lib64 -L/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/lib64 -Wl,@/sw/xc40cle6/darshan/3.1.5/sles12.3_gcc7.2.0/share/ld-opts/darshan-base-ld-opts -lfmpich -lmpichcxx
-Wl,--start-group -ldarshan -ldarshan-stubs -Wl,--end-group -lz -Wl,--no-as-needed,-lAtpSigHandler,-lAtpSigHCommData -Wl,--undefined=_ATP_Data_Globals -Wl,--undefined=__atpHandlerInstall -lpthread -lmpichf90_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mpi_mp
-lm -lf -lsci_cray_mp -lmpich_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mp -lcraymp -lm -lpthread -lf -lhugetlbfs -lpgas-dmapp -lfi -lu -lrt -Wl,--undefined=dmapp_get_flag_nbi -ldmapp -lugni -ludreg -lpthread -lm -lcray-c++-rts -lstdc++ -lxpmem -ldmapp -lpthread
-lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lugni -lpthread -lalpsutil -lpthread -lrca -ludreg -lquadmath -lm -lomp -lcraymp -lpthread -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lmodules -lm -lfi -lm -lquadmath -lcraymath -lm -lgfortran -lquadmath -lf -lm
-lpthread -lu -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lcsup -Wl,--as-needed,-latomic,--no-as-needed -Wl,--as-needed,-lgfortran,--no-as-needed -lstdc++ -lpthread -Wl,--start-group,-lc,-lcsup,-lgcc_eh,-lm,-lgcc,--end-group -Wl,-T/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/2.23.1.cce.ld <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/dmapp/default/lib64/libdmapp.a(dmapp_internal.o): In function `_dmappi_is_pure_dmapp_job':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/home/abuild/rpmbuild/BUILD/cray-dmapp-7.1.1/src/dmapp_internal.c:1401: undefined reference to `__wrap_MPI_Init'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_exit_.o): In function `blacs_exit_':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_exit_.c:42: undefined reference to `__wrap_MPI_Finalize'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.o): In function `blacs_pinfo_':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18: undefined reference to `__wrap_MPI_Init'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.oo): In function `Cblacs_pinfo':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18: undefined reference to `__wrap_MPI_Init'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/cce/8.6.5/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: link errors found, deleting executable `exe'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/usr/bin/sha1sum: exe: No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Darshan 3.1.5 darshan-runtime.pc<o:p></o:p></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"># This pkgconfig file was built for use with the Cray 2.x Programming</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"># Environment.</span></span><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Courier New",serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">It has not been tested for general purpose use.</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Name: darshan</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Description: Darshan runtime library</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Version: 3.1.5</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Requires: zlib mpich</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Requires.private:</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_prefix=/sw/xc40cle6/darshan/3.1.5/sles12.3_gcc7.2.0</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_share=/sw/xc40cle6/darshan/3.1.5/sles12.3_gcc7.2.0/share</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_includedir=</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_libdir= -L${darshan_prefix}/lib</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_linkopts="-Wl,@${darshan_share}/ld-opts/darshan-base-ld-opts"</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Cflags:</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Libs:</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Libs.private: ${darshan_linkopts} ${darshan_libdir} -lfmpich -lmpichcxx -Wl,--start-group -ldarshan -ldarshan-stubs -Wl,--end-group</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></span></p>
<p class="MsoNormal">Darshan 3.1.4 darshan-runtime.pc<o:p></o:p></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> more /sw/xc40cle6/darshan/3.1.4/sles12.3_gcc7.2.0/darshan-3.1.4/darshan-runtime/lib/pkgconfig/darshan-runtime.pc</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"># This pkgconfig file was built for use with the Cray 2.x Programming</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif"># Environment.</span></span><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Courier New",serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">It has not been tested for general purpose use.</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Name: darshan</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Description: Darshan runtime library</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Version: 3.1.4</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Requires: zlib mpich</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Requires.private:</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_prefix=/sw/xc40cle6/darshan/3.1.4/sles12.3_gcc7.2.0</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_share=/sw/xc40cle6/darshan/3.1.4/sles12.3_gcc7.2.0/share</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_includedir=</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_libdir= -L${darshan_prefix}/lib</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">darshan_linkopts="-Wl,@${darshan_share}/ld-opts/darshan-base-ld-opts"</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Cflags:</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Libs:</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Courier New",serif">Libs.private: ${darshan_linkopts} ${darshan_libdir} -lfmpich -lmpichcxx -Wl,--start-group -ldarshan -ldarshan-stubs -Wl,--end-group</span></span><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Darshan modules.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#%Module<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"># Module darshan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#module load texlive/2017<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">if { [ info exists env(PE_ENV) ] } {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> set PRGENV $env(PE_ENV)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">} else {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> set PRGENV "GNU"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">set PREFIX /sw/xc40cle6/darshan/3.1.5/sles12.3_gcc7.2.0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#PE_PKGCONFIG_LIBS contain a list of all pc files that apply to all supported languages<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">prepend-path PE_PKGCONFIG_LIBS darshan-runtime<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">prepend-path PKG_CONFIG_PATH $PREFIX/lib/pkgconfig
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">prepend-path LD_LIBRARY_PATH $PREFIX/lib
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">prepend-path PATH $PREFIX/bin
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">By the way, only 2 tests fails at compilations so far with darshan3.1.5, ( the above scalapack test) and a simple petsc code:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">cat petsc.c<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">static char help[] = "Solves 2D Poisson equation using multigrid.\n\n";<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#include <petscdm.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#include <petscdmda.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#include <petscksp.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#include <petscsys.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#include <petscvec.h><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">extern PetscErrorCode ComputeJacobian(KSP,Mat,Mat,void*);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">extern PetscErrorCode ComputeRHS(KSP,Vec,void*);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">extern PetscErrorCode ComputeTrueSolution(DM, Vec);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">extern PetscErrorCode VecView_VTK(Vec, const char [], const char []);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">typedef enum {DIRICHLET, NEUMANN} BCType;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">typedef struct {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscScalar uu, tt;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> BCType bcType;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">} UserContext;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#undef __FUNCT__<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#define __FUNCT__ "main"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">int main(int argc,char **argv)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">{<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> KSP ksp;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> DM da;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> UserContext user;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscInt bc;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscErrorCode ierr;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscInitialize(&argc,&argv,(char*)0,help);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDACreate2d(PETSC_COMM_WORLD, DM_BOUNDARY_NONE, DM_BOUNDARY_NONE,DMDA_STENCIL_STAR,-11,-11,PETSC_DECIDE,PETSC_DECIDE,1,1,NULL,NULL,&da);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPSetDM(ksp,(DM)da);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMSetApplicationContext(da,&user);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> user.uu = 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> user.tt = 1.0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> bc = (PetscInt)NEUMANN; <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> user.bcType = (BCType)bc;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPSetComputeRHS(ksp,ComputeRHS,&user);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPSetComputeOperators(ksp,ComputeJacobian,&user);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPSolve(ksp,NULL,NULL);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDestroy(&da);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPDestroy(&ksp);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = PetscFinalize();CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> return 0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#undef __FUNCT__<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#define __FUNCT__ "ComputeRHS"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">PetscErrorCode ComputeRHS(KSP ksp,Vec b,void *ctx)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">{<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> UserContext *user = (UserContext*)ctx;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscErrorCode ierr;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscInt i,j,M,N,xm,ym,xs,ys;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscScalar Hx,Hy,pi,uu,tt;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscScalar **array;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> DM da;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscFunctionBeginUser;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDAGetInfo(da, 0, &M, &N, 0,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> uu = user->uu; tt = user->tt;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> pi = 4*atan(1.0);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> Hx = 1.0/(PetscReal)(M);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> Hy = 1.0/(PetscReal)(N);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDAGetCorners(da,&xs,&ys,0,&xm,&ym,0);CHKERRQ(ierr); <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDAVecGetArray(da, b, &array);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> for (j=ys; j<ys+ym; j++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> for (i=xs; i<xs+xm; i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> array[j][i] = -PetscCosScalar(uu*pi*((PetscReal)i+0.5)*Hx)*PetscCosScalar(tt*pi*((PetscReal)j+0.5)*Hy)*Hx*Hy;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDAVecRestoreArray(da, b, &array);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = VecAssemblyBegin(b);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = VecAssemblyEnd(b);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (user->bcType == NEUMANN) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> MatNullSpace nullspace;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,0,&nullspace);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatNullSpaceRemove(nullspace,b);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatNullSpaceDestroy(&nullspace);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscFunctionReturn(0);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#undef __FUNCT__<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">#define __FUNCT__ "ComputeJacobian"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">PetscErrorCode ComputeJacobian(KSP ksp,Mat J, Mat jac,void *ctx)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">{<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> UserContext *user = (UserContext*)ctx;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscErrorCode ierr;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscInt i, j, M, N, xm, ym, xs, ys, num, numi, numj;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscScalar v[5], Hx, Hy, HydHx, HxdHy;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> MatStencil row, col[5];<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> DM da;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscFunctionBeginUser;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = KSPGetDM(ksp,&da);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDAGetInfo(da,0,&M,&N,0,0,0,0,0,0,0,0,0,0);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> Hx = 1.0 / (PetscReal)(M);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> Hy = 1.0 / (PetscReal)(N);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> HxdHy = Hx/Hy;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> HydHx = Hy/Hx;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = DMDAGetCorners(da,&xs,&ys,0,&xm,&ym,0);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> for (j=ys; j<ys+ym; j++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> for (i=xs; i<xs+xm; i++) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> row.i = i; row.j = j;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (i==0 || j==0 || i==M-1 || j==N-1) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (user->bcType == DIRICHLET) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Dirichlet boundary conditions not supported !\n");<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> else if (user->bcType == NEUMANN) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> num=0; numi=0; numj=0;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (j!=0) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[num] = -HxdHy; col[num].i = i; col[num].j = j-1;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> num++; numj++;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (i!=0) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[num] = -HydHx; col[num].i = i-1; col[num].j = j;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> num++; numi++;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (i!=M-1) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[num] = -HydHx; col[num].i = i+1; col[num].j = j;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> num++; numi++;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (j!=N-1) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[num] = -HxdHy; col[num].i = i; col[num].j = j+1;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> num++; numj++;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[num] = ((PetscReal)(numj)*HxdHy + (PetscReal)(numi)*HydHx); col[num].i = i; col[num].j = j;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> num++;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatSetValuesStencil(jac,1,&row,num,col,v,INSERT_VALUES);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> } else {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[0] = -HxdHy; col[0].i = i; col[0].j = j-1;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[1] = -HydHx; col[1].i = i-1; col[1].j = j;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[2] = 2.0*(HxdHy + HydHx); col[2].i = i; col[2].j = j;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[3] = -HydHx; col[3].i = i+1; col[3].j = j;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> v[4] = -HxdHy; col[4].i = i; col[4].j = j+1;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatSetValuesStencil(jac,1,&row,5,col,v,INSERT_VALUES);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatAssemblyBegin(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatAssemblyEnd(jac,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> if (user->bcType == NEUMANN) {<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> MatNullSpace nullspace;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatNullSpaceCreate(PETSC_COMM_WORLD,PETSC_TRUE,0,0,&nullspace);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatSetNullSpace(jac,nullspace);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> ierr = MatNullSpaceDestroy(&nullspace);CHKERRQ(ierr);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif"> PetscFunctionReturn(0);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">}<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> cc -o exe_petsc petsc.c -craype-verbose <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">driver.cc -hcpu=haswell -hstatic -D__CRAYXC -D__CRAY_HASWELL -D__CRAYXT_COMPUTE_LINUX_TARGET -hnetwork=aries -o exe_petsc petsc.c -Wl,-rpath=/opt/cray/pe/cce/8.6.5/cce/x86_64/lib
-hlast_user_arg -nostdinc -uno_driver_includes -isystem /opt/cray/pe/cce/8.6.5/cce/x86_64/include/craylibs -isystem /opt/cray/pe/cce/8.6.5/cce/x86_64/include/basic -isystem /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0/include -isystem /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0/include-fixed
-isystem /opt/gcc/6.1.0/snos/include -L/opt/gcc/6.1.0/snos/lib64 -Wl,-rpath=/opt/cray/pe/gcc-libs -L/usr/lib64 -L/lib64 -ugcc_base=/opt/gcc/6.1.0/snos -uno_driver_libs -isystem /usr/include -I/opt/cray/pe/petsc/3.7.6.2/real/CRAY/8.6/haswell/include -I/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/include
-I/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/include -I/opt/cray/pe/tpsl/17.11.1/CRAY/8.6/haswell/include -I/opt/cray/pe/hdf5-parallel/1.10.1.1/CRAY/8.6/include -I/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/include -I/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/include
-I/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari/include -I/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/include -I/opt/cray/pe/pmi/5.0.13/include -I/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/include
-I/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/include -I/opt/cray/krca/2.2.2-6.0.5.0_5.46__g4614cf3.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/pe/petsc/3.7.6.2/real/CRAY/8.6/haswell/lib
-L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib -L/opt/cray/pe/tpsl/17.11.1/CRAY/8.6/haswell/lib
-L/opt/cray/pe/hdf5-parallel/1.10.1.1/CRAY/8.6/lib -L/sw/xc40cle6/darshan/3.1.5/sles12.3_gcc7.2.0/lib -L/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/lib64 -L/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/lib64 -L/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/lib64
-L/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/lib64 -L/opt/cray/pe/pmi/5.0.13/lib64 -L/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/lib64 -L/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/lib64 -L/opt/cray/pe/atp/2.1.1/libApp -L/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/pkgconfig/../
-L/lib64 -L/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/lib64 -Wl,@/sw/xc40cle6/darshan/3.1.5/sles12.3_gcc7.2.0/share/ld-opts/darshan-base-ld-opts -lfmpich -lmpichcxx -Wl,--start-group -ldarshan -ldarshan-stubs -Wl,--end-group -lz -Wl,--no-as-needed,-lAtpSigHandler,-lAtpSigHCommData
-Wl,--undefined=_ATP_Data_Globals -Wl,--undefined=__atpHandlerInstall -lpthread -lcraypetsc_cray_real -homp -lm -lmodules -lf -lHYPRE_cray -lcmumps_cray -ldmumps_cray -lesmumps_cray -lmatio_cray -lptscotcherrexit_cray -lptscotchparmetis_cray -lscotcherrexit_cray
-lscotchmetis_cray -lsmumps_cray -lsundials_arkode_cray -lsundials_cvode_cray -lsundials_cvodes_cray -lsundials_fnvecopenmp_cray -lsundials_fnvecparallel_cray -lsundials_fnvecserial_cray -lsundials_ida_cray -lsundials_idas_cray -lsundials_kinsol_cray -lsuperlu_cray
-lsuperlu_dist_cray -lzmumps_cray -lsundials_nvecopenmp_cray -lsundials_nvecparallel_cray -lsundials_nvecserial_cray -lmumps_common_cray -lptscotch_cray -lparmetis_cray -lptesmumps_cray -lpord_cray -lmetis_cray -lscotch_cray -lptscotcherr_cray -lscotcherr_cray
-lcraymp -lpthread -lm -lomp64 -lf -lz -lrt -lhdf5_parallel -lrt -lz -ldl -lm -lpthread -lmpichf90_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mpi_mp -lm -lf -lsci_cray_mp -lmpich_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mp -lcraymp -lm -lpthread -lf -lhugetlbfs
-lpgas-dmapp -lfi -lu -lrt -Wl,--undefined=dmapp_get_flag_nbi -ldmapp -lugni -ludreg -lpthread -lm -lcray-c++-rts -lstdc++ -lxpmem -ldmapp -lpthread -lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lugni -lpthread -lalpsutil -lpthread -lrca -ludreg -lquadmath
-lm -lomp -lcraymp -lpthread -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lmodules -lm -lfi -lm -lquadmath -lcraymath -lm -lgfortran -lquadmath -lf -lm -lpthread -lu -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lcsup -Wl,--as-needed,-latomic,--no-as-needed -lstdc++ -lpthread
-Wl,--start-group,-lc,-lcsup,-lgcc_eh,-lm,-lgcc,--end-group -Wl,-T/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/2.23.1.cce.ld <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/hdf5-parallel/1.10.1.1/CRAY/8.6/lib/libhdf5_parallel.a(H5PL.o): In function `H5PL__open$$CFE_id_56395c9c_a2f1556b':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/hdf5-support/rpm/BUILD/cray-hdf5-1.10.1.1-201709131735.a6e5c5fe22823-cce1-parallel/hdf5-1.10.1/src/H5PL.c:878: warning: Using 'dlopen' in statically linked applications requires
at runtime the shared libraries from the glibc version used for linking<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/dmapp/default/lib64/libdmapp.a(dmapp_internal.o): In function `_dmappi_is_pure_dmapp_job':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/home/abuild/rpmbuild/BUILD/cray-dmapp-7.1.1/src/dmapp_internal.c:1401: undefined reference to `__wrap_MPI_Init'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/home/abuild/rpmbuild/BUILD/cray-dmapp-7.1.1/src/dmapp_internal.c:1401: undefined reference to `__wrap_MPI_Init_thread'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/petsc/3.7.6.2/real/CRAY/8.6/haswell/lib/libcraypetsc_cray_real.a(pinit.o): In function `PetscInitialize':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/cray-petsc/.cray-build/CRAY/86/haswell/cray-petsc/petsc-3.7.6/src/sys/objects/pinit.c:779: undefined reference to `__wrap_MPI_Init_thread'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/petsc/3.7.6.2/real/CRAY/8.6/haswell/lib/libcraypetsc_cray_real.a(pinit.o): In function `PetscFinalize':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/cray-petsc/.cray-build/CRAY/86/haswell/cray-petsc/petsc-3.7.6/src/sys/objects/pinit.c:1428: undefined reference to `__wrap_MPI_Finalize'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/tpsl/17.11.1/CRAY/8.6/haswell/lib/libHYPRE_cray.a(mpistubs.o): In function `hypre_MPI_Init':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/ulib/tpsl/.cray-build/CRAY/86/haswell/hypre/hypre-2.11.2/src/utilities/mpistubs.c:663: undefined reference to `__wrap_MPI_Init'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/tpsl/17.11.1/CRAY/8.6/haswell/lib/libHYPRE_cray.a(mpistubs.o): In function `hypre_MPI_Finalize':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/ulib/tpsl/.cray-build/CRAY/86/haswell/hypre/hypre-2.11.2/src/utilities/mpistubs.c:669: undefined reference to `__wrap_MPI_Finalize'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/tpsl/17.11.1/CRAY/8.6/haswell/lib/libparmetis_cray.a(gkmpi.c.o): In function `libparmetis__gkMPI_Finalize':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/ulib/tpsl/.cray-build/CRAY/86/haswell/parmetis/parmetis-4.0.2/libparmetis/gkmpi.c:288: undefined reference to `__wrap_MPI_Finalize'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.o): In function `blacs_pinfo_':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18: undefined reference to `__wrap_MPI_Init'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.oo): In function `Cblacs_pinfo':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18: undefined reference to `__wrap_MPI_Init'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/cce/8.6.5/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: link errors found, deleting executable `exe_petsc'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/usr/bin/sha1sum: exe_petsc: No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> module swap darshan/3.1.5 darshan/3.1.4 <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> cc -o exe_petsc petsc.c -craype-verbose <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">driver.cc -hcpu=haswell -hstatic -D__CRAYXC -D__CRAY_HASWELL -D__CRAYXT_COMPUTE_LINUX_TARGET -hnetwork=aries -o exe_petsc petsc.c -Wl,-rpath=/opt/cray/pe/cce/8.6.5/cce/x86_64/lib
-hlast_user_arg -nostdinc -uno_driver_includes -isystem /opt/cray/pe/cce/8.6.5/cce/x86_64/include/craylibs -isystem /opt/cray/pe/cce/8.6.5/cce/x86_64/include/basic -isystem /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0/include -isystem /opt/gcc/6.1.0/snos/lib/gcc/x86_64-suse-linux/6.1.0/include-fixed
-isystem /opt/gcc/6.1.0/snos/include -L/opt/gcc/6.1.0/snos/lib64 -Wl,-rpath=/opt/cray/pe/gcc-libs -L/usr/lib64 -L/lib64 -ugcc_base=/opt/gcc/6.1.0/snos -uno_driver_libs -isystem /usr/include -I/opt/cray/pe/petsc/3.7.6.2/real/CRAY/8.6/haswell/include -I/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/include
-I/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/include -I/opt/cray/pe/tpsl/17.11.1/CRAY/8.6/haswell/include -I/opt/cray/pe/hdf5-parallel/1.10.1.1/CRAY/8.6/include -I/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/include -I/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/include
-I/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/include -I/opt/cray/gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari/include -I/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/include -I/opt/cray/pe/pmi/5.0.13/include -I/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/include
-I/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/include -I/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/include -I/opt/cray/krca/2.2.2-6.0.5.0_5.46__g4614cf3.ari/include -I/opt/cray-hss-devel/8.0.0/include -L/opt/cray/pe/petsc/3.7.6.2/real/CRAY/8.6/haswell/lib
-L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/dmapp/default/lib64 -L/opt/cray/pe/mpt/7.7.0/gni/mpich-cray/8.6/lib -L/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib -L/opt/cray/pe/tpsl/17.11.1/CRAY/8.6/haswell/lib
-L/opt/cray/pe/hdf5-parallel/1.10.1.1/CRAY/8.6/lib -L/sw/xc40cle6/darshan/3.1.4/sles12.3_gcc7.2.0/lib -L/opt/cray/rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari/lib64 -L/opt/cray/alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari/lib64 -L/opt/cray/xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari/lib64
-L/opt/cray/dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari/lib64 -L/opt/cray/pe/pmi/5.0.13/lib64 -L/opt/cray/ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari/lib64 -L/opt/cray/udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari/lib64 -L/opt/cray/pe/atp/2.1.1/libApp -L/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/pkgconfig/../
-L/lib64 -L/opt/cray/wlm_detect/1.3.2-6.0.5.0_3.1__g388ccd5.ari/lib64 -Wl,@/sw/xc40cle6/darshan/3.1.4/sles12.3_gcc7.2.0/share/ld-opts/darshan-base-ld-opts -lfmpich -lmpichcxx -Wl,--start-group -ldarshan -ldarshan-stubs -Wl,--end-group -lz -Wl,--no-as-needed,-lAtpSigHandler,-lAtpSigHCommData
-Wl,--undefined=_ATP_Data_Globals -Wl,--undefined=__atpHandlerInstall -lpthread -lcraypetsc_cray_real -homp -lm -lmodules -lf -lHYPRE_cray -lcmumps_cray -ldmumps_cray -lesmumps_cray -lmatio_cray -lptscotcherrexit_cray -lptscotchparmetis_cray -lscotcherrexit_cray
-lscotchmetis_cray -lsmumps_cray -lsundials_arkode_cray -lsundials_cvode_cray -lsundials_cvodes_cray -lsundials_fnvecopenmp_cray -lsundials_fnvecparallel_cray -lsundials_fnvecserial_cray -lsundials_ida_cray -lsundials_idas_cray -lsundials_kinsol_cray -lsuperlu_cray
-lsuperlu_dist_cray -lzmumps_cray -lsundials_nvecopenmp_cray -lsundials_nvecparallel_cray -lsundials_nvecserial_cray -lmumps_common_cray -lptscotch_cray -lparmetis_cray -lptesmumps_cray -lpord_cray -lmetis_cray -lscotch_cray -lptscotcherr_cray -lscotcherr_cray
-lcraymp -lpthread -lm -lomp64 -lf -lz -lrt -lhdf5_parallel -lrt -lz -ldl -lm -lpthread -lmpichf90_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mpi_mp -lm -lf -lsci_cray_mp -lmpich_cray -lrt -lpthread -lugni -lpmi -lsci_cray_mp -lcraymp -lm -lpthread -lf -lhugetlbfs
-lpgas-dmapp -lfi -lu -lrt -Wl,--undefined=dmapp_get_flag_nbi -ldmapp -lugni -ludreg -lpthread -lm -lcray-c++-rts -lstdc++ -lxpmem -ldmapp -lpthread -lpmi -lpthread -lalpslli -lpthread -lwlm_detect -lugni -lpthread -lalpsutil -lpthread -lrca -ludreg -lquadmath
-lm -lomp -lcraymp -lpthread -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lmodules -lm -lfi -lm -lquadmath -lcraymath -lm -lgfortran -lquadmath -lf -lm -lpthread -lu -lrt -ldl -lcray-c++-rts -lstdc++ -lm -lcsup -Wl,--as-needed,-latomic,--no-as-needed -lstdc++ -lpthread
-Wl,--start-group,-lc,-lcsup,-lgcc_eh,-lm,-lgcc,--end-group -Wl,-T/opt/cray/pe/cce/8.6.5/cce/x86_64/lib/2.23.1.cce.ld <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/opt/cray/pe/hdf5-parallel/1.10.1.1/CRAY/8.6/lib/libhdf5_parallel.a(H5PL.o): In function `H5PL__open$$CFE_id_56395c9c_a2f1556b':<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">/b/hdf5-support/rpm/BUILD/cray-hdf5-1.10.1.1-201709131735.a6e5c5fe22823-cce1-parallel/hdf5-1.10.1/src/H5PL.c:878: warning: Using 'dlopen' in statically linked applications requires
at runtime the shared libraries from the glibc version used for linking<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New",serif">hadrib@cdl1:~> <o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sorry for the lengthy email :D<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Bilel <o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">-------<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Bilel Hadri, PhD<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Computational Scientist<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">KAUST Supercomputing Lab<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Al Khawarizmi Bldg. (1) Office 126<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">4700 King Abdullah University of Science and Technology<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Thuwal 23955-6900<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Kingdom of Saudi Arabia<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Office Phone: +966 12 808 0654<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Cell Phone: + 966 544 700 893<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Phil Carns <carns@mcs.anl.gov><br>
<b>Date: </b>Monday, February 26, 2018 at 8:18 PM<br>
<b>To: </b>"darshan-users@lists.mcs.anl.gov" <darshan-users@lists.mcs.anl.gov>, "bilel.hadri@kaust.edu.sa" <bilel.hadri@kaust.edu.sa><br>
<b>Subject: </b>Re: [Darshan-users] darshan3.1.5 issue on Cray XC40 cle6.up05<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal">Actually, it turns out that we don't have a up05 system to test on :)
<br>
<br>
Can you show us the output of the failing ftn command if you add the -craype-verbose command line option? It might also be helpful to see the contents of the darshan-runtime.pc and module file.<br>
<br>
We'll try building scalapack on some up04 systems to see if we either see the same problem or can tell what the environment difference is if not.<br>
<br>
thanks,<br>
-Phil<br>
<br>
On 02/26/2018 10:18 AM, Phil Carns wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi Bilel,<br>
<br>
Thanks for the bug report. We can try to reproduce here and confirm.<br>
<br>
In the mean time, can you tell me if the test_scalapack.f90 code is using MPI or not?
<br>
<br>
thanks,<br>
-Phil<br>
<br>
On 02/24/2018 02:25 PM, Bilel Hadri wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Dear Darshan colleagues,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I recently installed darshan 3.1.5 on Shaheen, Cray XC40, we recently upgraded the OS to CLE6.up05 and using the 17.12 PrgEnv.<o:p></o:p></p>
<p class="MsoNormal">Compiling Scalapack with Cray Libsci fails with the error shown below with all programming environment. Similar issue was observed for other codes, like a simple petsc code.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">After digging, it seems that it is related to darshan3.1.5 version recently installed on Shaheen. When unloading darshan, the compilation works fine with no issue.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">The error is not appearing when using darshan 3.1.4.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">ftn -o exe test_scalapack.f90 <o:p></o:p></p>
<p class="MsoNormal">/opt/cray/dmapp/default/lib64/libdmapp.a(dmapp_internal.o): In function `_dmappi_is_pure_dmapp_job':<o:p></o:p></p>
<p class="MsoNormal">/home/abuild/rpmbuild/BUILD/cray-dmapp-7.1.1/src/dmapp_internal.c:1401: undefined reference to `__wrap_MPI_Init'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_exit_.o): In function `blacs_exit_':<o:p></o:p></p>
<p class="MsoNormal">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_exit_.c:42: undefined reference to `__wrap_MPI_Finalize'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.o): In function `blacs_pinfo_':<o:p></o:p></p>
<p class="MsoNormal">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18: undefined reference to `__wrap_MPI_Init'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/CRAY/8.6/x86_64/lib/libsci_cray_mpi_mp.a(blacs_pinfo_.oo): In function `Cblacs_pinfo':<o:p></o:p></p>
<p class="MsoNormal">/b/worker/csml-libsci-sles/build/mp/scalapack/BLACS/SRC/blacs_pinfo_.c:18: undefined reference to `__wrap_MPI_Init'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/cce/8.6.5/binutils/x86_64/x86_64-pc-linux-gnu/bin/ld: link errors found, deleting executable `exe'<o:p></o:p></p>
<p class="MsoNormal">/usr/bin/sha1sum: exe: No such file or directory<o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> ll /usr/bin/sha1<o:p></o:p></p>
<p class="MsoNormal">sha1pass sha1sum <o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> ll /usr/bin/sha1sum <o:p></o:p></p>
<p class="MsoNormal">-rwxr-xr-x 1 root root 43912 Aug 6 2016 /usr/bin/sha1sum<o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> which sha1sum<o:p></o:p></p>
<p class="MsoNormal">/usr/bin/sha1sum<o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> module list <o:p></o:p></p>
<p class="MsoNormal">Currently Loaded Modulefiles:<o:p></o:p></p>
<p class="MsoNormal"> 1) modules/3.2.10.6 9) pmi/5.0.13 17) atp/2.1.1<o:p></o:p></p>
<p class="MsoNormal"> 2) eproxy/2.0.22-6.0.5.0_2.1__g1ebe45c.ari 10) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari 18) perftools-base/7.0.0<o:p></o:p></p>
<p class="MsoNormal"> 3) cce/8.6.5 11) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari 19) PrgEnv-cray/6.0.4<o:p></o:p></p>
<p class="MsoNormal"> 4) craype-network-aries 12) xpmem/2.2.4-6.0.5.0_4.8__g35d5e73.ari 20) cray-mpich/7.7.0<o:p></o:p></p>
<p class="MsoNormal"> 5) craype/2.5.13 13) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari 21) slurm/slurm<o:p></o:p></p>
<p class="MsoNormal"> 6) cray-libsci/17.12.1 14) dvs/2.7_2.2.52-6.0.5.2_17.6__g5170dea 22) craype-haswell<o:p></o:p></p>
<p class="MsoNormal"> 7) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari 15) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari 23) texlive/2017<o:p></o:p></p>
<p class="MsoNormal"> 8) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari 16) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari 24) darshan/3.1.5<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> module swap PrgEnv-cray/6.0.4 PrgEnv-intel<o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> ftn -o exe_i test_scalapack.f90 <o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/INTEL/16.0/x86_64/lib/libsci_intel_mpi.a(blacs_exit_.o): In function `blacs_exit_':<o:p></o:p></p>
<p class="MsoNormal">blacs_exit_.c:(.text+0xe9): undefined reference to `__wrap_MPI_Finalize'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/INTEL/16.0/x86_64/lib/libsci_intel_mpi.a(blacs_pinfo_.o): In function `blacs_pinfo_':<o:p></o:p></p>
<p class="MsoNormal">blacs_pinfo_.c:(.text+0x9b): undefined reference to `__wrap_MPI_Init'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/INTEL/16.0/x86_64/lib/libsci_intel_mpi.a(blacs_pinfo_.oo): In function `Cblacs_pinfo':<o:p></o:p></p>
<p class="MsoNormal">blacs_pinfo_.c:(.text+0x9b): undefined reference to `__wrap_MPI_Init'<o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> <o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> <o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> module swap PrgEnv-intel/6.0.4 PrgEnv-gnu<o:p></o:p></p>
<p class="MsoNormal">PrgEnv-gnu PrgEnv-gnu/6.0.4 <o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> module swap PrgEnv-intel/6.0.4 PrgEnv-gnu<o:p></o:p></p>
<p class="MsoNormal">hadrib@cdl1:~> ftn -o exe_i test_scalapack.f90 <o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/GNU/6.1/x86_64/lib/libsci_gnu_61_mpi.a(blacs_exit_.o): In function `blacs_exit_':<o:p></o:p></p>
<p class="MsoNormal">blacs_exit_.c:(.text+0xdb): undefined reference to `__wrap_MPI_Finalize'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/GNU/6.1/x86_64/lib/libsci_gnu_61_mpi.a(blacs_pinfo_.o): In function `blacs_pinfo_':<o:p></o:p></p>
<p class="MsoNormal">blacs_pinfo_.c:(.text+0xb3): undefined reference to `__wrap_MPI_Init'<o:p></o:p></p>
<p class="MsoNormal">/opt/cray/pe/libsci/17.12.1/GNU/6.1/x86_64/lib/libsci_gnu_61_mpi.a(blacs_pinfo_.oo): In function `Cblacs_pinfo':<o:p></o:p></p>
<p class="MsoNormal">blacs_pinfo_.c:(.text+0xb3): undefined reference to `__wrap_MPI_Init'<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">=====<o:p></o:p></p>
<p class="MsoNormal">implicit none<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">integer :: n, nb ! problem size and block size<o:p></o:p></p>
<p class="MsoNormal">integer :: myunit ! local output unit number<o:p></o:p></p>
<p class="MsoNormal">integer :: myArows, myAcols ! size of local subset of global array<o:p></o:p></p>
<p class="MsoNormal">integer :: i,j, igrid,jgrid, iproc,jproc, myi,myj, p<o:p></o:p></p>
<p class="MsoNormal">real*8, dimension(:,:), allocatable :: myA,myB,myC<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">integer :: numroc ! blacs routine<o:p></o:p></p>
<p class="MsoNormal">integer :: me, procs, icontxt, prow, pcol, myrow, mycol ! blacs data<o:p></o:p></p>
<p class="MsoNormal">integer :: info ! scalapack return value<o:p></o:p></p>
<p class="MsoNormal">integer, dimension(9) :: ides_a, ides_b, ides_c ! scalapack array desc<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">open(unit=1,file="ABCp.dat",status="old",form="formatted")<o:p></o:p></p>
<p class="MsoNormal">read(1,*)prow<o:p></o:p></p>
<p class="MsoNormal">read(1,*)pcol<o:p></o:p></p>
<p class="MsoNormal">read(1,*)n<o:p></o:p></p>
<p class="MsoNormal">read(1,*)nb<o:p></o:p></p>
<p class="MsoNormal">close(1)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">if (((n/nb) < prow) .or. ((n/nb) < pcol)) then<o:p></o:p></p>
<p class="MsoNormal">print *,"Problem size too small for processor set!"<o:p></o:p></p>
<p class="MsoNormal">stop 100<o:p></o:p></p>
<p class="MsoNormal">endif<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">call blacs_pinfo (me,procs)<o:p></o:p></p>
<p class="MsoNormal">call blacs_get (0, 0, icontxt)<o:p></o:p></p>
<p class="MsoNormal">call blacs_gridinit(icontxt, 'R', prow, pcol)<o:p></o:p></p>
<p class="MsoNormal">call blacs_gridinfo(icontxt, prow, pcol, myrow, mycol)<o:p></o:p></p>
<p class="MsoNormal">myunit = 10+me<o:p></o:p></p>
<p class="MsoNormal">write(myunit,*)"--------"<o:p></o:p></p>
<p class="MsoNormal">write(myunit,*)"Output for processor ",me," to unit ",myunit<o:p></o:p></p>
<p class="MsoNormal">write(myunit,*)"Proc ",me,": myrow, mycol in p-array is ", &<o:p></o:p></p>
<p class="MsoNormal">myrow, mycol<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">myArows = numroc(n, nb, myrow, 0, prow)<o:p></o:p></p>
<p class="MsoNormal">myAcols = numroc(n, nb, mycol, 0, pcol)<o:p></o:p></p>
<p class="MsoNormal">write(myunit,*)"Size of global array is ",n," x ",n<o:p></o:p></p>
<p class="MsoNormal">write(myunit,*)"Size of block is ",nb," x ",nb<o:p></o:p></p>
<p class="MsoNormal">write(myunit,*)"Size of local array is ",myArows," x ",myAcols<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">allocate(myA(myArows,myAcols))<o:p></o:p></p>
<p class="MsoNormal">allocate(myB(myArows,myAcols))<o:p></o:p></p>
<p class="MsoNormal">allocate(myC(myArows,myAcols))<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">do i=1,n<o:p></o:p></p>
<p class="MsoNormal">call g2l(i,n,prow,nb,iproc,myi)<o:p></o:p></p>
<p class="MsoNormal">if (myrow==iproc) then<o:p></o:p></p>
<p class="MsoNormal">do j=1,n<o:p></o:p></p>
<p class="MsoNormal">call g2l(j,n,pcol,nb,jproc,myj)<o:p></o:p></p>
<p class="MsoNormal">if (mycol==jproc) then<o:p></o:p></p>
<p class="MsoNormal">myA(myi,myj) = real(i+j)<o:p></o:p></p>
<p class="MsoNormal">myB(myi,myj) = real(i-j)<o:p></o:p></p>
<p class="MsoNormal">myC(myi,myj) = 0.d0<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">endif<o:p></o:p></p>
<p class="MsoNormal">enddo<o:p></o:p></p>
<p class="MsoNormal">endif<o:p></o:p></p>
<p class="MsoNormal">enddo<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">! Prepare array descriptors for ScaLAPACK<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">ides_a(1) = 1 ! descriptor type<o:p></o:p></p>
<p class="MsoNormal">ides_a(2) = icontxt ! blacs context<o:p></o:p></p>
<p class="MsoNormal">ides_a(3) = n ! global number of rows<o:p></o:p></p>
<p class="MsoNormal">ides_a(4) = n ! global number of columns<o:p></o:p></p>
<p class="MsoNormal">ides_a(5) = nb ! row block size<o:p></o:p></p>
<p class="MsoNormal">ides_a(6) = nb ! column block size<o:p></o:p></p>
<p class="MsoNormal">ides_a(7) = 0 ! initial process row<o:p></o:p></p>
<p class="MsoNormal">ides_a(8) = 0 ! initial process column<o:p></o:p></p>
<p class="MsoNormal">ides_a(9) = myArows ! leading dimension of local array<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">do i=1,9<o:p></o:p></p>
<p class="MsoNormal">ides_b(i) = ides_a(i)<o:p></o:p></p>
<p class="MsoNormal">ides_c(i) = ides_a(i)<o:p></o:p></p>
<p class="MsoNormal">enddo<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">! Call ScaLAPACK library routine<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">call pdgemm('T','T',n,n,n,1.0d0, myA,1,1,ides_a, &<o:p></o:p></p>
<p class="MsoNormal">myB,1,1,ides_b,0.d0, &<o:p></o:p></p>
<p class="MsoNormal">myC,1,1,ides_c )<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">! Print results<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">call g2l(n,n,prow,nb,iproc,myi)<o:p></o:p></p>
<p class="MsoNormal">call g2l(n,n,pcol,nb,jproc,myj)<o:p></o:p></p>
<p class="MsoNormal">if ((myrow==iproc) .and. (mycol==jproc)) &<o:p></o:p></p>
<p class="MsoNormal">write(*,*) 'c(',n,n,')=',myC(myi,myj)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">! Deallocate the local arrays<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">deallocate(myA, myB, myC)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">! End blacs for processors that are used<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">call blacs_gridexit(icontxt)<o:p></o:p></p>
<p class="MsoNormal">call blacs_exit(0)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">end<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">! convert global index to local index in block-cyclic distribution<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">subroutine g2l(i,n,np,nb,p,il)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">implicit none<o:p></o:p></p>
<p class="MsoNormal">integer :: i ! global array index, input<o:p></o:p></p>
<p class="MsoNormal">integer :: n ! global array dimension, input<o:p></o:p></p>
<p class="MsoNormal">integer :: np ! processor array dimension, input<o:p></o:p></p>
<p class="MsoNormal">integer :: nb ! block size, input<o:p></o:p></p>
<p class="MsoNormal">integer :: p ! processor array index, output<o:p></o:p></p>
<p class="MsoNormal">integer :: il ! local array index, output<o:p></o:p></p>
<p class="MsoNormal">integer :: im1 <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">im1 = i-1<o:p></o:p></p>
<p class="MsoNormal">p = mod((im1/nb),np)<o:p></o:p></p>
<p class="MsoNormal">il = (im1/(np*nb))*nb + mod(im1,nb) + 1<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">return<o:p></o:p></p>
<p class="MsoNormal">end<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">! convert local index to global index in block-cyclic distribution<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">subroutine l2g(il,p,n,np,nb,i)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">implicit none<o:p></o:p></p>
<p class="MsoNormal">integer :: il ! local array index, input<o:p></o:p></p>
<p class="MsoNormal">integer :: p ! processor array index, input<o:p></o:p></p>
<p class="MsoNormal">integer :: n ! global array dimension, input<o:p></o:p></p>
<p class="MsoNormal">integer :: np ! processor array dimension, input<o:p></o:p></p>
<p class="MsoNormal">integer :: nb ! block size, input<o:p></o:p></p>
<p class="MsoNormal">integer :: i ! global array index, output<o:p></o:p></p>
<p class="MsoNormal">integer :: ilm1 <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">ilm1 = il-1<o:p></o:p></p>
<p class="MsoNormal">i = (((ilm1/nb) * np) + p)*nb + mod(ilm1,nb) + 1<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">return<o:p></o:p></p>
<p class="MsoNormal">end<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">-------</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:black"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Bilel Hadri, PhD</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Computational Scientist</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">KAUST Supercomputing Lab</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Al Khawarizmi Bldg. (1) Office 126</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">4700 King Abdullah University of Science and Technology</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Thuwal 23955-6900</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Kingdom of Saudi Arabia</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Office Phone: +966 12 808 0654</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Cell Phone: + 966 544 700 893</span><o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11.0pt">
<hr size="2" width="100%" align="center">
</span></div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:gray">This message and its contents including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message
in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.<br>
</span><span style="font-size:11.0pt"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Darshan-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Darshan-users@lists.mcs.anl.gov">Darshan-users@lists.mcs.anl.gov</a><o:p></o:p></pre>
<pre><a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a><o:p></o:p></pre>
</blockquote>
<p><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Darshan-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Darshan-users@lists.mcs.anl.gov">Darshan-users@lists.mcs.anl.gov</a><o:p></o:p></pre>
<pre><a href="https://lists.mcs.anl.gov/mailman/listinfo/darshan-users">https://lists.mcs.anl.gov/mailman/listinfo/darshan-users</a><o:p></o:p></pre>
</blockquote>
<p><o:p> </o:p></p>
</div>
</body>
</html>