[petsc-dev] could we let "make -j N" work for PETSc and external packages?

[Fande Kong]

On Wed, Jul 25, 2018 at 9:16 AM, Satish Balay <balay at mcs.anl.gov> wrote:

> With what compilers? Is this license issue - or RAM issue? [i.e what
> errors do you get?]
>

I do not have these details any more. Definitely will send you the
information next time when we hit this issue.


>
> If your compiler fails this way - its likely to affect externalpackage
> builds aswell. So you might need to use configure option:
>
> --with-make-np=1
>

Right, sometime we have to use this to build external packages as well.


>
> [But I would think a value higher than 1 should work..]
>


Honestly, I did not try anything higher than 1 when we were hitting  the
issue.  "1" always works for us.

Fande,


>
> Satish
>
> On Wed, 25 Jul 2018, Fande Kong wrote:
>
> > The default value determined by the configure sometimes makes the
> compiling process fail.  I have experienced a couple of times before. The
> solution is to use "MAKE_NP=1".
> >
> >
> >
> > Fande,
> >
> >
> > ________________________________
> > From: petsc-dev <petsc-dev-bounces at mcs.anl.gov> on behalf of Satish
> Balay <balay at mcs.anl.gov>
> > Sent: Tuesday, July 24, 2018 5:57:55 PM
> > To: Fande Kong
> > Cc: petsc-dev
> > Subject: Re: [petsc-dev] could we let "make -j N" work for PETSc and
> external packages?
> >
> > Another alternative: use gmakefile directly - and ignore our
> > 'makefile' wrapper.
> >
> > [with makefile wrapper - we try to control a few things - and print
> > some extra debug info like the MAKEFLAGS].
> >
> > But if you use 'gmakefile' directly - you would be using PETSc like
> > any other package - it supports 'V=1' for some debug info [which is
> > similar to info provided by most packages]
> >
> >
> > BTW: Any particular why MAKE_NP determined by configure [i.e the default
> value] is not useful?
> >
> > Satish
> >
> > On Tue, 24 Jul 2018, Fande Kong wrote:
> >
> > > Thanks, Satish,
> > >
> > > I’m going to use MAKE_NP
> > >
> > > Fande
> > >
> > > Sent from my iPhone
> > >
> > > > On Jul 24, 2018, at 4:26 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > > >
> > > > I hate html e-mail which doesn't add reply markers - can't figure
> out easily what you saying :(
> > > >
> > > >>> However some older make version might not display this info
> properly. [it should use the user specified '-j' value though]
> > > >>
> > > >> This confused me, and now I understood.  But it is misleading.
> Could we do something at the PETSc side regardless the make version? At
> least display the user-specified value not the default value.  I thought my
> "-j" did not change anything.
> > > >
> > > > Thats a 'make' issue. you are specifying options to make. So we have
> no control on it [unless make tells us - and this mode is buggy on the OSX
> default make version].
> > > >
> > > > The only option we have is : not print anything. I think thats a bad
> option.
> > > >
> > > > You have alternatives:
> > > >
> > > > - use a newer make with --download-make
> > > >
> > > > - or use MAKE_NP
> > > >
> > > > Satish
> > > >
> > > >> On Tue, 24 Jul 2018, Fande Kong wrote:
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________
> > > >> From: Satish Balay <balay at mcs.anl.gov>
> > > >> Sent: Tuesday, July 24, 2018 4:06 PM
> > > >> To: Fande Kong
> > > >> Cc: Jed Brown; For users of the development version of PETSc
> > > >> Subject: Re: [petsc-dev] could we let "make -j N" work for PETSc
> and external packages?
> > > >>
> > > >> PETSc configure sets up a default value that always gets used.
> > > >>
> > > >>>>>>>>>
> > > >> balay at asterix /home/balay/tmp/petsc (maint=)
> > > >> $ make
> > > >> <snip>
> > > >> Using MAKEFLAGS: -j4 -l6.0 w -- PETSC_DIR=/home/balay/tmp/petsc
> PETSC_ARCH=arch-linux2-c-debug
> > > >> <<<
> > > >>
> > > >> You can always override this default by adding a -j option. [since
> the
> > > >> user specified -j value comes at the end - it overrides the one that
> > > >> is specified at the begining of make command]
> > > >>
> > > >>>>>>>>>>>>
> > > >> balay at asterix /home/balay/tmp/petsc (maint=)
> > > >> $ make -j20
> > > >> <snip>
> > > >> Using MAKEFLAGS: -j4 -l6.0 w -j20 --jobserver-auth=3,4 --
> PETSC_DIR=/home/balay/tmp/petsc PETSC_ARCH=arch-linux2-c-debug
> > > >> <<<<<<<<<<
> > > >>
> > > >>
> > > >> However some older make version might not display this info
> properly. [it should use the user specified '-j' value though]
> > > >>
> > > >> This confused me, and now I understood.  But it is misleading.
> Could we do something at the PETSc side regardless the make version? At
> least display the user-specified value not the default value.  I thought my
> "-j" did not change anything.
> > > >>
> > > >> Fande,
> > > >>
> > > >>>>>>>>>
> > > >> [petsc at ipro petsc.maint ((879a78556f...))]$ make -j2
> > > >> Using MAKEFLAGS: -j7 -l12.0 w --jobserver-fds=3,4 -j --
> PETSC_DIR=/Users/petsc/petsc.maint PETSC_ARCH=arch-osx-10.6
> > > >> <<<<<<<
> > > >>
> > > >> You can use the alternate mode - just modify the default '-j' value
> used MAKE_NP
> > > >>
> > > >>>>>>>>>>
> > > >> [petsc at ipro petsc.maint ((879a78556f...))]$ make MAKE_NP=2
> > > >> Using MAKEFLAGS: -j2 -l12.0 w -- PETSC_DIR=/Users/petsc/petsc.maint
> PETSC_ARCH=arch-osx-10.6 MAKE_NP=2
> > > >> <<<<<<<
> > > >>
> > > >> Or you can control this default value at configure time [this
> affects externalpackage builds]
> > > >>
> > > >> ./configure --with-make-np=2
> > > >>
> > > >> Satish
> > > >>
> > > >>
> > > >>> On Tue, 24 Jul 2018, Fande Kong wrote:
> > > >>>
> > > >>> Thanks, Jed and Satish,
> > > >>>
> > > >>>
> > > >>> But "make -j N" does not change anything when compiling petsc.
> > > >>>
> > > >>>
> > > >>>
> > > >>> If I do something like: "make -j whatever". Then I always get:
> > > >>>
> > > >>>
> > > >>> Using MAKEFLAGS: -j8 -l36.0 w
> > > >>>
> > > >>>
> > > >>> It looks like PETSc always uses "-j something" that we can not
> change using the command-line.
> > > >>>
> > > >>>
> > > >>> Is it a bug?
> > > >>>
> > > >>>
> > > >>>
> > > >>> Fande,
> > > >>>
> > > >>>
> > > >>>
> > > >>> ________________________________
> > > >>> From: Jed Brown <jed at jedbrown.org>
> > > >>> Sent: Tuesday, July 24, 2018 3:44 PM
> > > >>> To: Fande Kong; For users of the development version of PETSc
> > > >>> Subject: Re: [petsc-dev] could we let "make -j N" work for PETSc
> and external packages?
> > > >>>
> > > >>> Fande Kong <Fande.Kong at inl.gov> writes:
> > > >>>
> > > >>>> Hi Developers,
> > > >>>>
> > > >>>>
> > > >>>> Could we somehow let "make -j N" to take multiple threads for
> compiling PETSc. Do we use multiple threads when compiling the external
> packages such as hypre, parmetis etc.?
> > > >>>
> > > >>> Yes, see self.parallelMake=0 for the few packages that don't
> support parallel make.
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180725/2fb5fb9f/attachment-0001.html>