[petsc-dev] make -j not supported? Why???

[Karl Rupp]

Hi,

>    Is it still not possible to use cmake to build cuda stuff? That's crazy?

It is possible, but you cannot specify nvcc as CMAKE_CXX_COMPILER, but 
instead the script is doing some magic hackery replacing the 
C++-compiler with nvcc if the file extension is .cu.

(I consider this to be a hack rather than a clean solution, but that's 
the status quo)

Best regards,
Karli

>
> On Feb 25, 2013, at 5:06 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>
>> Parallel build prints someting like:
>>
>>>>>>>>>>
>> ==========================================
>> Building PETSc using CMake with 5 build threads
>> ==========================================
>> Re-run cmake file: Makefile older than: ../CMakeLists.txt
>> -- Configuring done
>> -- Generating done
>> -- Build files have been written to: /home/balay/spetsc/asterix64
>> Scanning dependencies of target petsc
>> [  0%] Building C object CMakeFiles/petsc.dir/src/sys/classes/draw/utils/lgc.c.o
>> [  1%] Building C object CMakeFiles/petsc.dir/src/mat/impls/mffd/mffd.c.o
>> [  1%] [  1%] Building C object CMakeFiles/petsc.dir/src/dm/impls/plex/ftn-auto/plexcreatef.c.o
>>
>> <<<<<<<<<
>>
>> etc..
>>
>> Note: if you are building cuda stuff or have python < 2.5 - then you
>> are using legacy build [i.e sequential/recursive make] and see stuff
>> like:
>>
>>>>>>>>
>> libfast in: /home/balay/spetsc/src
>> libfast in: /home/balay/spetsc/src/sys
>> libfast in: /home/balay/spetsc/src/sys/classes
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer/impls
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer/impls/socket
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer/impls/socket/ftn-custom
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer/impls/ascii
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer/impls/ascii/ftn-auto
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer/impls/ascii/ftn-custom
>> libfast in: /home/balay/spetsc/src/sys/classes/viewer/impls/binary
>> <<<<
>> etc..
>>
>>
>> Satish
>>
>> On Mon, 25 Feb 2013, Nystrom, William D wrote:
>>
>>> Is there a way to examine the petsc build logs and determine if I am really getting parallel builds?  I have
>>> a top level script that I use to the petsc configure script followed by make.  For a couple of our test bed
>>> systems that stay fairly unloaded, I use the following configure option: --with-make-np=16.  But I also
>>> invoke my script like this: /usr/bin/time -p makePetscDev_caddy >& makePetscDev_caddy.log.  And
>>> I'll get a time like the following:
>>>
>>> real 2160.77
>>> user 1308.21
>>> sys 234.37
>>>
>>> So it does not seem like I am getting much if any parallelism in my build.  I am also building with lots
>>> of external packages and have no idea if parallel builds are being done with them.
>>>
>>> Thanks,
>>>
>>> Dave
>>>
>>> --
>>> Dave Nystrom
>>> LANL HPC-5
>>> Phone: 505-667-7913
>>> Email: wdn at lanl.gov
>>> Smail: Mail Stop B272
>>>        Group HPC-5
>>>        Los Alamos National Laboratory
>>>        Los Alamos, NM 87545
>>>
>>>
>>> ________________________________________
>>> From: petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] on behalf of Barry Smith [bsmith at mcs.anl.gov]
>>> Sent: Monday, February 25, 2013 3:25 PM
>>> To: For users of the development version of PETSc
>>> Subject: Re: [petsc-dev] make -j not supported? Why???
>>>
>>>   Tim,
>>>
>>>    As usual, a too short error message without enough information. I've changed it so that it will now give you
>>>
>>> Barrys-MacBook-Pro:petsc-dev barrysmith$ make -j 3 ccmake
>>> ********************* ERROR *************************
>>> PETSc compiles are automatically parallel, do not
>>> provide the -j option to make
>>> ******************************************************
>>> make: *** [chk_makej] Error 1
>>> Barrys-MacBook-Pro:petsc-dev barrysmith$ make -j 3 all-legacy
>>> ********************* ERROR *************************
>>> Parallel build with 'make -j' is not supported
>>> for PETSc legacy builds. Run without -j <np>
>>> or ./configure PETSc with --download-cmake
>>> to enable parallel builds with PETSc
>>> ******************************************************
>>> make[1]: *** [chk_makej] Error 1
>>> make: *** [all-legacy] Error 2
>>> Barrys-MacBook-Pro:petsc-dev barrysmith$
>>>
>>> Explanations:
>>> When PETSc is using cmake makefiles from the top level makefiles we pass our own -j <np> down, make doesn't like having two levels of -j and at best prints a confusing warning message about the parallel build.
>>>
>>> When using PETSc's legacy make system it uses a recursive make that is not able to be parallelized (yes it is bad and hence the cmake version is used by default now).
>>>
>>> Notes:
>>> It would be nice if we could somehow remove the -j <np> at the top level so that it didn't generate the error message and just ran but make doesn't allow this type of thing.
>>>
>>> It would be nice if we could eliminate the messages like
>>>
>>> make[1]: *** [chk_makej] Error 1
>>> make: *** [all-legacy] Error 2
>>>
>>> that make generates automatically but I cannot figure out how to do that.
>>>
>>> So the current situation is a compromise where most users get automatically a parallel build and the users who knowingly provide a -j <np> option get slightly peeved. (My guess is 90+% of people who install PETSc don't know about the -j option).  At least now it will explain that it is doing a parallel build.
>>>
>>> I'll check on the -k option next.
>>>
>>>   Barry
>>>
>>> On Feb 25, 2013, at 2:58 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>>>
>>>> This is to support different backends. If configure found CMake, then normal "make" will use multiple jobs, or you can do it explicitly with make -j3 $PETSC_ARCH.
>>>>
>>>> CMake would be terrible as a hard dependency (it's terrible enough as it is) but it's less bad than some alternatives.
>>>>
>>>>
>>>> On Mon, Feb 25, 2013 at 3:30 PM, Tim Tautges <tautges at mcs.anl.gov> wrote:
>>>> tautges at chronos:~/code/petsc-dev-moab$ make -j 8
>>>> ********************* ERROR *************************
>>>> Parallel build with 'make -j' is not supported
>>>> ******************************************************
>>>> make: *** [chk_makej] Error 1
>>>> tautges at chronos:~/code/petsc-dev-moab$
>>>>
>>>>
>>>> Why??? I can almost understand petsc's justification for its own autotools-looking python-based build system, but why its need to not support otherwise-standard make options?  Similarly, PETSC_MAKE_STOP_ON_ERROR instead of make -k?
>>>>
>>>> - tim
>>>>
>>>> --
>>>> ================================================================
>>>> "You will keep in perfect peace him whose mind is
>>>>   steadfast, because he trusts in you."               Isaiah 26:3
>>>>
>>>>              Tim Tautges            Argonne National Laboratory
>>>>          (tautges at mcs.anl.gov)      (telecommuting from UW-Madison)
>>>> phone (gvoice): (608) 354-1459      1500 Engineering Dr.
>>>>             fax: (608) 263-4499      Madison, WI 53706
>>>>
>>>>
>>>
>>>
>>
>