[petsc-dev] Sean is going to love this

[Geoff Oxberry]

On Wed, Dec 31, 2014 at 11:40 AM, Geoff Oxberry <goxberry at gmail.com> wrote:

>
>
> On Tue, Dec 30, 2014 at 10:40 AM, Sean Farley <
> sean.michael.farley at gmail.com> wrote:
>
>>
>> Barry Smith writes:
>>
>> >> On Dec 29, 2014, at 9:56 PM, Geoff Oxberry <goxberry at gmail.com> wrote:
>> >>
>> >> > Date: Sun, 28 Dec 2014 21:27:21 -0600
>> >> > From: Barry Smith <bsmith at mcs.anl.gov>
>> >> > To: Sean Farley <sean.michael.farley at gmail.com>
>> >> > Cc: petsc-dev <petsc-dev at mcs.anl.gov>
>> >> > Subject: Re: [petsc-dev] Sean is going to love this
>> >> > Message-ID: <BA03ABA6-C693-4404-9FA5-2211BC371836 at mcs.anl.gov>
>> >> > Content-Type: text/plain; charset="utf-8"
>> >> >
>> >> >
>> >> >   Sean,
>> >> >
>> >> >     brew install /homebrew/science/petsc
>> >> >     brew install /homebrew/science/petsc --HEAD --with-x11
>> >> >
>> >> >   Is there any reason not to use home-brew for everything now?
>> Should we be working with the homebrew/science guys to work out the rough
>> edges? (Like their hypre build is sequential and they don't pass the MPI
>> compiler wrappers properly to build packages and ... and ...)
>> >>
>> >> I contributed most of the hypre recipe; brew install hypre --with-mpi
>> should get you a parallel build, and it should also run some parallel
>> exampled as smoke tests. If there's anything missing that should be added,
>> let me know and I can get to it when I return from vacation.
>> >
>> >    Geoff,
>> >
>> >     Thanks, I noticed your name on a bunch of things there. I may
>> bother you later with some issues when I understand brew better but, what
>> the heck, here are a few things I noticed. (By the way, overall I was
>> impressed with brew working with all these "nasty" scientific packages)
>>
>> No love for the work I did with all the nasty scientific packages? Oh,
>> ok :-(
>>
>> > 1)   I ran brew install /homebrew/science/petsc --HEAD --with-hypre and
>> it failed because it just used the bottled hypre (which is sequential) as a
>> dependent package and then tried to build the parallel PETSc with the
>> sequential hypre so failed in the middle of compiling PETSc because of
>> inconsistent hypre include files (ones for sequential).
>>
>>
> Brew bottles were originally only used in situations where building from
> source would take a long time, and under the assumption that most users are
> interested in a "standard" build without any command-line options.
> Recently, this idea seems to have been extended to most brew formula. I
> don't think brew bottles make sense for something like hypre, which is
> relatively quick to build, because the convenience savings are negligible.
>
>
>> Yep, this is a problem that doesn't exist for PETSc in MacPorts.
>>
>> >    Presumably had I done a brew install hypre --with-mpi first and then
>> the brew install /homebrew/science/petsc --HEAD it may have worked
>> properly? (but how is a naive new user to know?)
>>
>> Indeed.
>>
>
> You're right. The PETSc formula should require that if hypre is used, it
> must have been built with the "--with-mpi" flag. This issue would have
> tripped me up also. I will add a GitHub issue for this point:
> https://github.com/Homebrew/homebrew-science/issues/1602
>

Dominique Orban (@dpo) has submitted a pull request that should remedy this
issue: https://github.com/Homebrew/homebrew-science/pull/1604

He's also added some commentary responding to Barry's e-mail re: brew; some
of the material is covered in my earlier response, and he raises some
additional interesting points:
https://github.com/Homebrew/homebrew-science/issues/1602#issuecomment-68472966


>
> >    This seems to be related to my first meta issue which is how is brew
>> really suppose to work with versions of packages (and build option versions
>> for dependent packages)? For example petsc 3.5.2 uses certain versions of
>> other packages such as mumps, superlu_dist etc but petsc --HEAD (which will
>> become PETSc 3.6) may work with later versions of some of the packages. How
>> is this suppose to be indicated, handled, etc? I couldn't see if brew had
>> the concept of handling multiple versions of packages and different version
>> dependencies for each version?
>
> Brew formulas can express package dependencies that include install flags;
>  https://github.com/Homebrew/homebrew/blob/master/Library/Contributions/example-formula.rb
> <https://github.com/Homebrew/homebrew/blob/master/Library/Contributions/example-formula.rb> is
> a comprehensive example of what can and can't be done.
>
> Indicating version dependencies is problematic. The brew package model
> assumes users want the latest versions of packages, which has led to sets
> of formulas like {python, python3} in the main homebrew repo, and {metis,
> metis4} in cases where packages depend on specific versions. So brew does
> not have a sophisticated way of handling packages with multiple version
> numbers.
>
> If one has used brew to install different versions of a package over time,
> such as installing `open-mpi` version 1.7.4, then later upgrading to
> version 1.8.1, the `brew-switch` command can be used to switch among
> versions (e.g., `brew switch open-mpi 1.7.4` or `brew switch open-mpi
> 1.8.1` in this example, depending on which version is currently active).
> `brew info` can be used to query the currently active version (e.g., `brew
> info open-mpi`).
>
> If PETSc only uses the latest versions of packages, the best way forward
> is probably to update the formulas for PETSc dependencies. If PETSc is tied
> to specific package versions, then it might be better to have a separate
> PETSc tap, with namespaced dependencies that don't conflict with
> homebrew-science. Dominique Orban was working on a FEniCS tap for a while (
> https://github.com/optimizers/homebrew-fenics); I had mixed success when
> using it.
>
> Since Sean is going to say it if I don't, yes, I think MacPorts does a
> better job indicating version dependencies, and Homebrew has room for
> improvement here.
>
>>
>> >    Second meta issue, is there a way to force a compiled of the package
>> instead of it using the prepared bottle?
>
> Or even better to just tell brew I never want it to use bottles but always
>> want it to compile for all packages? One reason I ask this is because PETSc
>> has so many optional dependencies it seems maybe? better to just have it
>> build based on the choices the user made rather than using some pre-built
>> bottle that only has certain things turned on? But maybe I don't understand
>> bottles and when they are used instead of building.
>>
>
>
> https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/FAQ.md#why-do-you-compile-everything
>
> To summarize, a formula will build from source if:
>
> - any installation option is added (e.g., `brew install petsc --HEAD`)
> - the environment variable HOMEBREW_BUILD_FROM_SOURCE is set
> - the option `--build-from-source` is invoked (e.g., `brew install
> --build-from-source hypre`)
>
>
>> This is done in MacPorts.
>>
>> > 2) I think PETSc is just "lucky" that it builds at all. brew seems to
>> set the environmental variables CC, CXX, and FC to clang, clang++ and
>> gfortran but then PETSc actually ignores these variables (printing a
>> warning if you run with -vd) and hunts for mpicc, mpicxx, mpifc which it
>> happens to find in /usr/local/bin since open-mpi is a dependency of PETSc
>> and thus must have been installed first.
>>
>> This is why sandboxing is important for package managers.
>>
>
> I agree with Sean here.
>
> Brew sets a bunch of environmental variables in something it calls
> superenv; the rationale is given in
> https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Homebrew-0.9.3.md.
> Much of the code for the sandboxing is in
> https://github.com/Homebrew/homebrew/tree/master/Library/Homebrew/extend,
> in `ENV.rb` or the `ENV` directory.
>
>
>> >    So this leads the third meta issue, it seems brew doesn't seem to
>> have any concept of some packages requiring mpi wrapper compilers to be
>> used to compile the package and so it is kind of catch as catch can if MPI
>> based packages will build properly instead of having a systematic way of
>> handling it. For example if PETSc did not ignore CC, CXX, and FC  it would
>> try to use clang, clang++ and gfortran and then crap out that it didn't
>> have MPI compilers. Any chance that brew would ever be smart enough to
>> "reset" the compilers to the MPI ones before building packages that depend
>> on MPI?
>>
>
> I think the next step is to improve support for MPI wrapper compilers
> within formulas of packages that depend on MPI or have MPI options. I can
> look at MacPorts and BuildSystem for inspiration here to see what could be
> done.
>
>
>>
>> A hard graph problem that I hacked in MacPorts. I have no idea about how
>> to do it in brew (hence why I haven't used it).
>>
>> > 3) PETSc has a bunch of other optional dependencies such as hdf5,
>> netcdf ... that it would be nice to support. Presumably I could try to make
>> a pull request with support for them but I'm concerned that my lack of
>> understanding of meta issue two might mess me up.
>>
>> All of those are supported in MacPorts.
>>
>
> As long as formulas exist (or are created) for these optional
> dependencies, I think adding support for them is doable, and probably
> straightforward.
>
>
>>
>> > 4) The issue of MPICH2 and open-mpi conflicting is a pain. Some package
>> managers handle this by having a concept of a meta-package or
>> abstract-package (such as MPI) that can have multiple implementation. Then,
>> for example, one could say PETSc is dependent on MPI (and not open-mpi or
>> MPICH) and the package system would work properly if either open-mpi or
>> MPICH had already been installed (instead of crapping out like it does if I
>> first install MPICH and then install PETSc; since it tries to install
>> open-mpi for PETSc which conflicts with MPICH and thus stops).
>>
>> Also, this is done in MacPorts.
>>
>
> Brew does not handle open-mpi vs MPICH well. An inelegant solution that
> would probably work would be to hardcode paths for each MPI implementation.
> If the only implementations used are open-mpi and MPICH, this solution is
> expedient and probably fine. I started along this path for MUMPS in a pull
> request (https://github.com/Homebrew/homebrew-science/pull/771) that
> brought up relevant discussion with Jed, Aron, and Dominique Orban.
>
> One issue arising with brew is that if a package like `mumps` is build
> with `mpich2`, and then `mpich2` is uninstalled and replaced with
> `open-mpi`, brew should rebuild `mumps`. Right now, it won't force a
> rebuild like that. Also, the `:mpi` dependency only resolves to `open-mpi`.
> I think it would be worth revisiting this issue to point out the demand for
> better MPI dependency tracking and resolution.
>
>
>>
>> With that being said, I must also declare that I have no love for
>> MacPorts. It gets the job done for these specific cases and works just
>> well enough but still has other problems. MacPorts will never be able to
>> solve harder DAG problems for dependencies (just hacks). I wish I had
>> the time to build a new package manager, but alas. The quote from ESR
>> seems to ring true:
>>
>
> MacPorts is okay. Sean's put a lot of work into it, which I appreciate,
> and I've used MacPorts in the past. I eventually found it hard to maintain
> because duplicating system programs into their own sandbox required
> environment hacking to make sure I got the right versions on my PATH. I
> found it more difficult to maintain. Since Homebrew avoids duplicating
> system utilities (e.g., compilers), I find it easier to maintain a cleaner
> system setup.
>
> I also found it difficult to contribute to MacPorts. I'd rather use Git
> and Ruby over SVN and TCL; in both cases, the former technology is more
> widely used than the latter. Getting a pull request accepted as a new
> contributed to Homebrew was easy; the same did not appear to be true for
> MacPorts. I think these reasons might contribute to why MacPorts has 194
> contributors (https://www.openhub.net/p/MacPorts), and Homebrew has 5,034
> contributors (https://www.openhub.net/p/homebrew). Last I checked,
> Homebrew was the 12th-most starred and 9th-most forked repo on GitHub.
>
> From a feature standpoint, I think MacPorts does a lot of things
> feature-wise that Homebrew should be doing, so I take it as a guide for how
> to improve Homebrew.
>
>
>>
>> The most dangerous enemy of a better solution is an existing codebase
>> that is just good enough.
>>
>
>
> --
> Geoffrey Oxberry, Ph.D., E.I.T.
> goxberry at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20141231/a6683633/attachment.html>