<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerimcfxmj { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonimcfxmj:hover, .pfptPrimaryButtonimcfxmj:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonimcfxmj:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Satish, Many thanks for the interesting response. I'll see if I can figure out something useful to do for our goals (a reliable spack package for our simulator, which is dependent on bringing in the exact right petsc build as dependency),
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerimcfxmj" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerimcfxmj" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerimcfxmj" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerimcfxmj" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div>Hi Satish,</div><div><br></div><div>Many thanks for the interesting response. I'll see if I can figure out something useful to do for our goals (a reliable spack package for our simulator, which is dependent on bringing in</div><div>the exact right petsc build as dependency), but as you say, maintaining these things and accounting for everything is a lot harder than it looks. Time constraints from other projects</div><div>might limit how much more tinkering I can do, but we have a better idea of the issues we face now.</div><div><br></div><div>Daniel<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 24, 2024 at 4:25 PM Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">This is the complexity with maintaining dependencies (and dependencies<br>
of dependencies), and different build systems<br>
<br>
- Its not easy to keep the "defaults" in both builds exactly the same.<br>
- And its not easy to expose all "variants" or keep the same variants in both builds.<br>
- And each pkg has its own issues that prevents some combinations to<br>
work or not [or tested combinations vs untested].<br>
<br>
This e-mail query has multiple things:<br>
<br>
- understand "why" the current impl of [spack, petsc] build tools are the way they are.<br>
- if they can be improved<br>
- and build use cases that you need working<br>
- [and subsequently your code working]<br>
<br>
Addressing them all is not easy - so lets stick with what you need to make progress.<br>
<br>
For one - we recommend using latest petsc version [i.e 3.21 - not 3.19] - any fixes we have will address the current release.<br>
<br>
> - spack: ptscotch will always be built without parmetis wrappers, can't turn on<br>
<br>
diff --git a/var/spack/repos/builtin/packages/petsc/package.py b/var/spack/repos/builtin/packages/petsc/package.py<br>
index b7b1d86b15..ae27ba4c4e 100644<br>
--- a/var/spack/repos/builtin/packages/petsc/package.py<br>
+++ b/var/spack/repos/builtin/packages/petsc/package.py<br>
@@ -268,9 +268,7 @@ def check_fortran_compiler(self):<br>
depends_on("metis@5:~int64", when="@3.8:+metis~int64")<br>
depends_on("metis@5:+int64", when="@3.8:+metis+int64")<br>
<br>
- # PTScotch: Currently disable Parmetis wrapper, this means<br>
- # nested disection won't be available thought PTScotch<br>
- depends_on("scotch+esmumps~metis+mpi", when="+ptscotch")<br>
+ depends_on("scotch+esmumps+mpi", when="+ptscotch")<br>
depends_on("scotch+int64", when="+ptscotch+int64")<br>
<br>
depends_on("hdf5@:1.10+mpi", when="@:3.12+hdf5+mpi")<br>
<br>
Now you can try:<br>
<br>
spack install petsc~metis+ptscotch ^scotch+metis<br>
vs<br>
spack install petsc~metis+ptscotch ^scotch~metis [~metis is the default for scotch]<br>
<br>
Note the following comment in spack/var/spack/repos/builtin/packages/scotch/package.py<br>
<br>
>>>><br>
# Vendored dependency of METIS/ParMETIS conflicts with standard<br>
# installations<br>
conflicts("metis", when="+metis")<br>
conflicts("parmetis", when="+metis")<br>
<<<<<<br>
<br>
> - classical: ptscotch will always be built with parmetis wrappers, can't seem to turn off<br>
<br>
Looks like spack uses cmake build of ptscotch. PETSc uses Makefile interface. It likely doesn't support turning off metis wrappers [without hacks].<br>
<br>
So you might either need to hack scotch build via petsc - or just install it separately - and use it with petsc.<br>
<br>
I see an effort to migrate scotch build in petsc to cmake<br>
<br>
<a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7242/__;!!G_uCfscf7eWS!axLWsOWdCnVQSjIurDkWvFmG4riOizNNPbVM78TQoVScHx7ERMUENiQ-VW2Lh5e83QHhKcA7-HO0nDJ_hTez8JffqQz8h1I$" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/7242/</a><br>
<a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7495/__;!!G_uCfscf7eWS!axLWsOWdCnVQSjIurDkWvFmG4riOizNNPbVM78TQoVScHx7ERMUENiQ-VW2Lh5e83QHhKcA7-HO0nDJ_hTez8JffgEb5IwI$" rel="noreferrer" target="_blank">https://gitlab.com/petsc/petsc/-/merge_requests/7495/</a><br>
<br>
Satish<br>
<br>
On Wed, 24 Apr 2024, Daniel Stone wrote:<br>
<br>
> Hi PETSc community,<br>
> <br>
> I've been looking at using Spack to build PETSc, in particular I need to<br>
> disable the default metis/parmetis dependencies and use PTScotch instead,<br>
> for our software.<br>
> I've had quite a bit of trouble with this - it seems like something in the<br>
> resulting build of our simulator ends up badly optimised and an mpi<br>
> bottleneck, when I build against<br>
> PETSc built with Spack.<br>
> <br>
> I've been trying to track this down, and noticed this in the PETSc Spack<br>
> build recipe:<br>
> <br>
> # PTScotch: Currently disable Parmetis wrapper, this means<br>
> # nested disection won't be available thought PTScotch<br>
> depends_on("scotch+esmumps~metis+mpi", when="+ptscotch")<br>
> depends_on("scotch+int64", when="+ptscotch+int64")<br>
> <br>
> <br>
> Sure enough - when I compare the build with Spack and a traditional build<br>
> with ./configure etc, I see that, in the traditional build, Scotch is<br>
> always built with the parmetis wrapper,<br>
> but not in the Scotch build. In fact, I'm not sure how to turn off the<br>
> parmetis wrapper option for scotch, in the case of a traditional build<br>
> (i.e. there doesn't seem to be a flag in the<br>
> configure script for it) - which would be a very useful test for me (I can<br>
> of course do similar experiments by doing a classical build of petsc<br>
> against ptscotch built separately without the<br>
> wrappers, etc - will try that).<br>
> <br>
> Does anyone know why the parmetis wrapper is always disabled in the spack<br>
> build options? Is there something about Spack that would prevent it from<br>
> working? It's notable - but I might<br>
> be missing it - that there's no warning that there's a difference in the<br>
> way ptscotch is built between the spack and classical builds:<br>
> - classical: ptscotch will always be built with parmetis wrappers, can't<br>
> seem to turn off<br>
> - spack: ptscotch will always be built without parmetis wrappers, can't<br>
> turn on<br>
> <br>
> Any insight at all would be great, I'm new to Spack and am not super<br>
> familiar with the logic that goes into setting up builds for the system.<br>
> <br>
> Here is the kind of command I give to Spack for PETSc builds, which may<br>
> well be less than ideal:<br>
> <br>
> spack install petsc@3.19.1 ~metis +ptscotch ^hdf5 +fortran +hl<br>
> <br>
> Separate tiny note: when building with hdf5, I have to ensure that the<br>
> fortran flag is set for it, as above. There's a fortran flag for the petsc<br>
> module, default true, and a fortran flag for the hdf5<br>
> module, default false. A naive user (i.e. me), will see the fortran flag<br>
> for the petsc module, and assume that all dependencies will correspondingly<br>
> be built with fortran capability - then see that<br>
> hdf5.mod is missing when trying to build their software against petsc. It's<br>
> the old "did you forget --with-hdf5-fortran-bindings?" issue, resurrected<br>
> for a new build system.<br>
> <br>
> Thanks,<br>
> <br>
> Daniel<br>
> <br>
<br>
</blockquote></div>