[petsc-dev] slow "ar Scq" on cygwin

[Chetan Jhurani]

> -----Original Message-----
> From: Satish Balay
> Sent: Thursday, June 28, 2012 7:39 PM
> To: For users of the development version of PETSc
> Subject: Re: [petsc-dev] slow "ar Scq" on cygwin
> 
> Hm - eventhough you added 'argparse.py' to python-2.6 on cygwin to get
> builder.py working - perhaps Matt is using some other python-2.7
> feature - and thats causing this difference?

Don't know.  For what it's worth, my Linux box has
python 2.6.6 and I had to put argparse.py there as
well.

Chetan

> 
> Satish
> 
> >
> >
> >
> > I don't mind extra debugging on cygwin/Windows before reporting issues
> >
> > since I know the petsc project has many other priorities. But a few useful
> >
> > debugging pointers will be serve both of us better.
> >
> >
> >
> > Chetan
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From: petsc-dev-bounces at mcs.anl.gov [mailto:petsc-dev-bounces at mcs.anl.gov] On Behalf Of Matthew
> Knepley
> > Sent: Thursday, June 28, 2012 6:00 PM
> > To: For users of the development version of PETSc
> > Subject: Re: [petsc-dev] slow "ar Scq" on cygwin
> >
> >
> >
> > On Thu, Jun 28, 2012 at 4:49 PM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> >
> > > > $ PETSC_DIR=/cygdrive/d/petsc-dev-2012-06-26 PETSC_ARCH=test_8 ./config/builder.py
> > > > (The same happens (existing .o are deleted) if ./config/builder2.py build is used)
> >
> > > This is not what I wrote. You used builder.py, which is python 2.4 compatible, but has to do
> everything without options,
> > > so yes it deletes all the objects and rebuilds everything. That is why I did not suggest you use
> it.
> >
> > > I said to use builder2.py, which does proper dependency tracking.
> >
> >
> >
> > That is what I used and I mentioned it above. I've tried both separately.
> >
> >
> >
> > Both give the same result that libpetsc.lib is not created.  And
> >
> > whether I use builder.py or builder2.py, the existing objects
> >
> > files are deleted in the second build. I did not call for a cleanup.
> >
> >
> >
> > Look. It does not do that. Ever. In any universe. I don't know what you think you are doing,
> >
> > but it is definitely not what you wrote. Look at the code. It can't do that.
> >
> >
> >
> >   matt
> >
> >
> >
> > To be sure, I tried the whole thing in a new ARCH with a fresh
> >
> > configure step, and this time builder.py was not used at all (by me)
> >
> > but it led to the same issues with builder2.py.
> >
> >
> >
> > I understand that this is not the most used platform, so let me
> >
> > know if I can be of help.  And I forgot to mention that the
> >
> > python build was 4-5 times faster than "ar" based build and
> >
> > that's great...
> >
> >
> >
> > Chetan
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > From: petsc-dev-bounces at mcs.anl.gov [mailto:petsc-dev-bounces at mcs.anl.gov] On Behalf Of Matthew
> Knepley
> > Sent: Thursday, June 28, 2012 3:21 PM
> > To: For users of the development version of PETSc
> > Subject: Re: [petsc-dev] slow "ar Scq" on cygwin
> >
> >
> >
> > On Thu, Jun 28, 2012 at 4:10 PM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> >
> > Matt,
> >
> >
> >
> > I was surprised too on both counts. Couldn't find any libpetsc.lib after the python build
> >
> > and I did not run "clean" in between that could have deleted the .o files.
> >
> >
> >
> > Here's one more try below to show it.
> >
> >
> >
> > $ ls test_8/lib/
> >
> > libpetsc-obj
> >
> >
> >
> > $ ls test_8/lib/libpetsc-obj/ | wc -l
> >
> > 560
> >
> >
> >
> > $ PETSC_DIR=/cygdrive/d/petsc-dev-2012-06-26 PETSC_ARCH=test_8 ./config/builder.py
> >
> > (The same happens (existing .o are deleted) if ./config/builder2.py build is used)
> >
> >
> >
> > This is not what I wrote. You used builder.py, which is python 2.4 compatible, but has to do
> everything without options,
> >
> > so yes it deletes all the objects and rebuilds everything. That is why I did not suggest you use it.
> >
> >
> >
> > I said to use builder2.py, which does proper dependency tracking.
> >
> >
> >
> >    Matt
> >
> >
> >
> >
> >
> > Rebuilding Dependencies
> >
> > Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys
> >
> > Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys/bag
> >
> > Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys/comm
> >
> > Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys/dll
> >
> >
> >
> > I stopped the build using control-c.  It would have deleted all files first.
> >
> >
> >
> > Traceback (most recent call last):
> >
> >   File "./config/builder.py", line 1534, in <module>
> >
> >     PETScMaker().run()
> >
> >   File "./config/builder.py", line 1526, in run
> >
> >     if self.buildLibraries('libpetsc', self.rootDir):
> >
> >   File "./config/builder.py", line 1331, in buildLibraries
> >
> >     objects += self.buildDir(root, files, objDir)
> >
> >   File "./config/builder.py", line 1251, in buildDir
> >
> >     objects.extend(getattr(self, 'compile'+language)(sourceMap[language], objDir))
> >
> >   File "./config/builder.py", line 1025, in compileC
> >
> >     return self.compile(self.configInfo.languages.clanguage, source, objDir)
> >
> >   File "./config/builder.py", line 1012, in compile
> >
> >     (output, error, status) = self.executeShellCommand(cmd, checkCommand = noCheckCommand,
> log=self.log)
> >
> >   File "/cygdrive/d/petsc-dev-2012-06-26/config/BuildSystem/script.py", line 254, in
> executeShellCommand
> >
> >     (output, error, status) = runInShell(command, log, cwd)
> >
> >   File "/cygdrive/d/petsc-dev-2012-06-26/config/BuildSystem/script.py", line 243, in runInShell
> >
> >     thread.join(timeout)
> >
> >   File "/usr/lib/python2.6/threading.py", line 655, in join
> >
> >     self.__block.wait(delay)
> >
> >   File "/usr/lib/python2.6/threading.py", line 258, in wait
> >
> >     _sleep(delay)
> >
> > KeyboardInterrupt
> >
> >
> >
> > $ ls test_8/lib/libpetsc-obj/
> >
> > bag.o
> >
> >
> >
> > All the previous .o were deleted.  Perhaps I'm not doing something right?
> >
> >
> >
> > Chetan
> >
> >
> >
> >
> >
> > From: petsc-dev-bounces at mcs.anl.gov [mailto:petsc-dev-bounces at mcs.anl.gov] On Behalf Of Matthew
> Knepley
> > Sent: Thursday, June 28, 2012 2:46 PM
> > To: For users of the development version of PETSc
> > Subject: Re: [petsc-dev] slow "ar Scq" on cygwin
> >
> >
> >
> > On Thu, Jun 28, 2012 at 3:31 PM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> >
> > Thanks Matt, Satish.  A few issues though.
> >
> >
> >
> > Like you said, the latest cygwin has python 2.6.8 which
> >
> > does not have argparse.  When I place the following
> >
> > in path http://argparse.googlecode.com/svn/trunk/argparse.py
> >
> > I can run the python build with 2.6.8.
> >
> >
> >
> > However, the "ar" based build had worked fine and the python
> >
> > based build fails for these two files.
> >
> >
> >
> >   src/snes/impls/MULTIB~1/multiblock.c
> >
> >   src/snes/impls/vi/rsaug/virsaug.c
> >
> >
> >
> > Yep, it fails, but the build succeeds, so no problem.
> >
> >
> >
> > It looks like these two files were NOT compiled in the ar based
> >
> > build. The two make.log files are attached.
> >
> >
> >
> > One more question. I tried to run the python based build
> >
> > the second time (after it was unsuccessful the first time)
> >
> > and then it deleted all the previous .o files it had created in
> >
> > ARCH/lib/libpetsc-obj.  How can that be avoided?
> >
> >
> >
> > It does not do that unless you run 'clean'.
> >
> >
> >
> >    Matt
> >
> >
> >
> > Chetan
> >
> >
> >
> >
> >
> > From: petsc-dev-bounces at mcs.anl.gov [mailto:petsc-dev-bounces at mcs.anl.gov] On Behalf Of Matthew
> Knepley
> > Sent: Thursday, June 28, 2012 1:35 PM
> > To: For users of the development version of PETSc
> > Subject: Re: [petsc-dev] slow "ar Scq" on cygwin
> >
> >
> >
> > On Thu, Jun 28, 2012 at 2:23 PM, Chetan Jhurani <chetan.jhurani at gmail.com> wrote:
> >
> > Hi devs,
> >
> > Is there a way to avoid running the archiver (ar)
> > every time a petsc directory is compiled?
> >
> > The reason I'm asking this is that when libpetsc.lib
> > becomes large enough (say 100MB+, esp. for debug build),
> > "ar Sqc" on cygwin takes very long in each directory.
> > Depending on OS caching, very long can be 10 seconds per
> > PETSc directory just to run ar.
> >
> > It creates a new temp file from scratch, puts stuff from
> > libpetsc.lib and new stuff in it and then renames the
> > temp file.  As you can imagine, the compilation process is
> > fast in the beginning but crawls by the end.  I've seen
> > the same IO sequence on Linux (with Sqc flags to ar), but
> > since cygwin IO is slower the effect is more clearly visible.
> > Building within a ramdisk leads to 25% faster compilation
> > but it was more of a curiosity.
> >
> >
> >
> > Build with Python. It never uses antique things like ar :)
> >
> >
> >
> >   python2.7 ./config/builder2.py build
> >
> >
> >
> >      Matt
> >
> >
> >
> >
> > Chetan
> >
> >
> >
> >
> >
> >
> >
> >