[petsc-dev] slow "ar Scq" on cygwin

[Matthew Knepley]

On Thu, Jun 28, 2012 at 8:26 PM, Chetan Jhurani <chetan.jhurani at gmail.com>wrote:

> > Look. It does not do that. Ever. In any universe. I don't know what you
> think you are doing,****
>
> > but it is definitely not what you wrote. Look at the code. It can't do
> that.****
>
> ** **
>
> I don’t think you understand what your code is doing in a new****
>
> environment of the same universe.
>

Yes, you are correct. I do not understand this code. I was wrong to tell
you to use it. Don't use it.

    Matt


>
>
> It works fine on Linux with the steps you had given.****
>
> But on the cygwin side, no dependency files like****
>
> ./lib/libpetsc-obj/adaptbasic.d are created. I found this after****
>
> running the python build on Linux (I didn’t know before this****
>
> where the dependencies were stored).****
>
> ** **
>
> Here is some debug info in the python code:****
>
> ** **
>
> In the buildLibraries function on the Linux box (when building in the
> second step)****
>
> ** **
>
>   self.sourceDatabase =  DirectedGraph with 1172 vertices and 21894 edges*
> ***
>
> ** **
>
> However, on the cygwin side (again, when building the second time)****
>
> ** **
>
>  self.sourceDatabase =  <builder.NullSourceDatabase object at 0xffd833ac>*
> ***
>
> ** **
>
> So of course it is totalRebuild = true on cygwin.****
>
> ** **
>
> I don’t mind extra debugging on cygwin/Windows before reporting issues****
>
> since I know the petsc project has many other priorities. But a few useful
> ****
>
> debugging pointers will be serve both of us better.****
>
> ** **
>
> Chetan****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> *From:* petsc-dev-bounces at mcs.anl.gov [mailto:
> petsc-dev-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley
> *Sent:* Thursday, June 28, 2012 6:00 PM
>
> *To:* For users of the development version of PETSc
> *Subject:* Re: [petsc-dev] slow "ar Scq" on cygwin****
>
> ** **
>
> On Thu, Jun 28, 2012 at 4:49 PM, Chetan Jhurani <chetan.jhurani at gmail.com>
> wrote:****
>
> *> > $ PETSC_DIR=/cygdrive/d/petsc-dev-2012-06-26 PETSC_ARCH=test_8
> ./config/builder.py
> > > (The same happens (existing .o are deleted) if ./config/builder2.py
> build is used)*****
>
> *> *This is not what I wrote. You used builder.py, which is python 2.4
> compatible, but has to do everything without options,
> *> *so yes it deletes all the objects and rebuilds everything. That is
> why I did not suggest you use it.****
>
> *> *I said to use builder2.py, which does proper dependency tracking.****
>
>  ****
>
> That is what I used and I mentioned it above. I’ve tried both separately.*
> ***
>
>  ****
>
> Both give the same result that libpetsc.lib is not created.  And****
>
> whether I use builder.py or builder2.py, the existing objects****
>
> files are deleted in the second build. I did not call for a cleanup.****
>
> ** **
>
> Look. It does not do that. Ever. In any universe. I don't know what you
> think you are doing,****
>
> but it is definitely not what you wrote. Look at the code. It can't do
> that.****
>
> ** **
>
>   matt****
>
>  ****
>
> To be sure, I tried the whole thing in a new ARCH with a fresh****
>
> configure step, and this time builder.py was not used at all (by me)****
>
> but it led to the same issues with builder2.py.****
>
>  ****
>
> I understand that this is not the most used platform, so let me****
>
> know if I can be of help.  And I forgot to mention that the****
>
> python build was 4-5 times faster than “ar” based build and****
>
> that’s great...****
>
>  ****
>
> Chetan****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> *From:* petsc-dev-bounces at mcs.anl.gov [mailto:
> petsc-dev-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley
> *Sent:* Thursday, June 28, 2012 3:21 PM
> *To:* For users of the development version of PETSc
> *Subject:* Re: [petsc-dev] slow "ar Scq" on cygwin****
>
>  ****
>
> On Thu, Jun 28, 2012 at 4:10 PM, Chetan Jhurani <chetan.jhurani at gmail.com>
> wrote:****
>
> Matt,****
>
>  ****
>
> I was surprised too on both counts. Couldn’t find any libpetsc.lib after
> the python build****
>
> and I did not run “clean” in between that could have deleted the .o files.
> ****
>
>  ****
>
> Here’s one more try below to show it.****
>
>  ****
>
> *$ ls test_8/lib/*****
>
> libpetsc-obj****
>
>  ****
>
> *$ ls test_8/lib/libpetsc-obj/ | wc -l*****
>
> *560*****
>
>  ****
>
> *$ PETSC_DIR=/cygdrive/d/petsc-dev-2012-06-26 PETSC_ARCH=test_8
> ./config/builder.py*****
>
> *(The same happens (existing .o are deleted) if ./config/builder2.py
> build is used)*****
>
>  ****
>
> This is not what I wrote. You used builder.py, which is python 2.4
> compatible, but has to do everything without options,****
>
> so yes it deletes all the objects and rebuilds everything. That is why I
> did not suggest you use it.****
>
>  ****
>
> I said to use builder2.py, which does proper dependency tracking.****
>
>  ****
>
>    Matt****
>
>  ****
>
> * *****
>
> Rebuilding Dependencies****
>
> Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys****
>
> Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys/bag****
>
> Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys/comm****
>
> Building in /cygdrive/d/petsc-dev-2012-06-26/src/sys/dll****
>
>  ****
>
> *I stopped the build using control-c.  It would have deleted all files
> first.*****
>
>  ****
>
> Traceback (most recent call last):****
>
>   File "./config/builder.py", line 1534, in <module>****
>
>     PETScMaker().run()****
>
>   File "./config/builder.py", line 1526, in run****
>
>     if self.buildLibraries('libpetsc', self.rootDir):****
>
>   File "./config/builder.py", line 1331, in buildLibraries****
>
>     objects += self.buildDir(root, files, objDir)****
>
>   File "./config/builder.py", line 1251, in buildDir****
>
>     objects.extend(getattr(self, 'compile'+language)(sourceMap[language],
> objDir))****
>
>   File "./config/builder.py", line 1025, in compileC****
>
>     return self.compile(self.configInfo.languages.clanguage, source,
> objDir)****
>
>   File "./config/builder.py", line 1012, in compile****
>
>     (output, error, status) = self.executeShellCommand(cmd, checkCommand =
> noCheckCommand, log=self.log)****
>
>   File "/cygdrive/d/petsc-dev-2012-06-26/config/BuildSystem/script.py",
> line 254, in executeShellCommand****
>
>     (output, error, status) = runInShell(command, log, cwd)****
>
>   File "/cygdrive/d/petsc-dev-2012-06-26/config/BuildSystem/script.py",
> line 243, in runInShell****
>
>     thread.join(timeout)****
>
>   File "/usr/lib/python2.6/threading.py", line 655, in join****
>
>     self.__block.wait(delay)****
>
>   File "/usr/lib/python2.6/threading.py", line 258, in wait****
>
>     _sleep(delay)****
>
> KeyboardInterrupt****
>
>  ****
>
> *$ ls test_8/lib/libpetsc-obj/*****
>
> *bag.o*****
>
>  ****
>
> All the previous .o were deleted.  Perhaps I’m not doing something right?*
> ***
>
>  ****
>
> Chetan****
>
>  ****
>
>  ****
>
> *From:* petsc-dev-bounces at mcs.anl.gov [mailto:
> petsc-dev-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley
> *Sent:* Thursday, June 28, 2012 2:46 PM
> *To:* For users of the development version of PETSc
> *Subject:* Re: [petsc-dev] slow "ar Scq" on cygwin****
>
>  ****
>
> On Thu, Jun 28, 2012 at 3:31 PM, Chetan Jhurani <chetan.jhurani at gmail.com>
> wrote:****
>
> Thanks Matt, Satish.  A few issues though.****
>
>  ****
>
> Like you said, the latest cygwin has python 2.6.8 which****
>
> does not have argparse.  When I place the following****
>
> in path http://argparse.googlecode.com/svn/trunk/argparse.py****
>
> I can run the python build with 2.6.8.****
>
>  ****
>
> However, the “ar” based build had worked fine and the python****
>
> based build fails for these two files.****
>
>  ****
>
>   src/snes/impls/MULTIB~1/multiblock.c****
>
>   src/snes/impls/vi/rsaug/virsaug.c****
>
>  ****
>
> Yep, it fails, but the build succeeds, so no problem.****
>
>  ****
>
> It looks like these two files were NOT compiled in the ar based****
>
> build. The two make.log files are attached.****
>
>  ****
>
> One more question. I tried to run the python based build****
>
> the second time (after it was unsuccessful the first time)****
>
> and then it deleted all the previous .o files it had created in ****
>
> ARCH/lib/libpetsc-obj.  How can that be avoided?****
>
>  ****
>
> It does not do that unless you run 'clean'.****
>
>  ****
>
>    Matt****
>
>  ****
>
> Chetan****
>
>  ****
>
>  ****
>
> *From:* petsc-dev-bounces at mcs.anl.gov [mailto:
> petsc-dev-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley
> *Sent:* Thursday, June 28, 2012 1:35 PM
> *To:* For users of the development version of PETSc
> *Subject:* Re: [petsc-dev] slow "ar Scq" on cygwin****
>
>  ****
>
> On Thu, Jun 28, 2012 at 2:23 PM, Chetan Jhurani <chetan.jhurani at gmail.com>
> wrote:****
>
> Hi devs,
>
> Is there a way to avoid running the archiver (ar)
> every time a petsc directory is compiled?
>
> The reason I'm asking this is that when libpetsc.lib
> becomes large enough (say 100MB+, esp. for debug build),
> "ar Sqc" on cygwin takes very long in each directory.
> Depending on OS caching, very long can be 10 seconds per
> PETSc directory just to run ar.
>
> It creates a new temp file from scratch, puts stuff from
> libpetsc.lib and new stuff in it and then renames the
> temp file.  As you can imagine, the compilation process is
> fast in the beginning but crawls by the end.  I've seen
> the same IO sequence on Linux (with Sqc flags to ar), but
> since cygwin IO is slower the effect is more clearly visible.
> Building within a ramdisk leads to 25% faster compilation
> but it was more of a curiosity.****
>
>  ****
>
> Build with Python. It never uses antique things like ar :)****
>
>  ****
>
>   python2.7 ./config/builder2.py build****
>
>  ****
>
>      Matt****
>
>  ****
>
>
> Chetan****
>
>
>
> ****
>
>  ****
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener****
>
>
>
> ****
>
>  ****
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener****
>
>
>
> ****
>
>  ****
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener****
>
>
>
> ****
>
> ** **
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener****
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120629/6b9a15ea/attachment.html>