[petsc-dev] Bash tool for handling switching between multiple PETSc branches and configures
Smith, Barry F.
bsmith at mcs.anl.gov
Fri Jan 3 20:06:35 CST 2020
I think they just have the wrong algorithms in their compilers for modules, I don't think there is anything fundamental to the language etc that requires that portion of the build to be so slow. Also gmake doesn't do a good job of making it easy to get overlap between building the Fortran modules and compiling the C; in some experiments I could get some overlap and better times but it was cumbersome so I gave up.
Barry
> On Jan 3, 2020, at 6:09 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> Same for me. There seems to be no way to get around this. Fortran is horrible, and hurts everyone all the time.
>
> Matt
>
> On Fri, Jan 3, 2020 at 7:03 PM <jed at jedbrown.org> wrote:
> The time to rebuild Fortran modules, which is pretty much an entire lifetime. I disable Fortran in most arches that I rebuild frequently.
>
> On Jan 3, 2020 16:58, "Smith, Barry F." <bsmith at mcs.anl.gov> wrote:
>
> And it you don't skip Fortran modules?
>
>
>
> > On Jan 3, 2020, at 4:36 PM, jed at jedbrown.org wrote:
> >
> > A few seconds if you skip Fortran modules.
> >
> > I didn't realize there were any active developers not using ccache. We should add a note to the manual.
> >
> > On Jan 3, 2020 15:04, "Smith, Barry F." <bsmith at mcs.anl.gov> wrote:
> >
> >
> > > On Jan 3, 2020, at 3:07 PM, Matthew Knepley <knepley at gmail.com> wrote:
> > >
> > > On Fri, Jan 3, 2020 at 3:52 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> > >
> > > A long time ago Oana suggested a tool that allowed switching between PETSc configurations after pulls etc that didn't require waiting to recompile code or rerun configure. Based on her ideas I finally got something that has been behaving reasonably satisfactory for me. Note that it is only useful if you work with PETSc in a way that you have multiple branches and PETSc arches you are dealing with. If you are essentially always in one branch it offers no advantages. (Unfortunately since Fortran stubs are stored in the source for branch changes that
> > > change the stubs the will require recompiles). This is my first public release so may have glitches. It saves me tons of time.
> > >
> > > Barry, I had the same problem, but it went away when I started using ccache in front of the compiler for every arch. Does
> > > this not work for you?
> >
> > I don't know, never tried it. Still it must "compile" the code (that is generate all the .o) and rebuild the libraries which takes some time?
> >
> > Barry
> >
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > > Put this in your .bashrc file and run source ~/.bashrc Docs are at the bottom.
> > >
> > > #
> > > # Utility function used by gitcheckout bash function
> > > # uses the basename of the branch to build the PETSC_ARCH
> > > #
> > > function branchtoarch () {
> > > basename `echo $1 | sed s?/maint?-maint?g` | sed -e s?/?-?g -e s?^?arch-?g
> > > }
> > >
> > > #
> > > # Used by the git alias provided below; not used directly
> > > #
> > > # Implements a git checkout, but preserves any previously compiled (.o) code so it won't unneccessariy get recompiled
> > > #
> > > # Sets PETSC_ARCH automatically based on the branch name
> > > #
> > > # How it works:
> > > # 1) before changing from the current PETSC_ARCH branch combination it deletes all out-dated .o files for that PETSC_ARCH (using make --dry-run to find these)
> > > # 2) after checking into the new PETSC_ARCH branch combination it updates the time-stamps of all remaing .o files for the new PETSC_ARCH so they are
> > > # more recent than any timestamps on the source files that git may have changed (it can do this because it knows from 1) that any out-of-date
> > > # .o files have already been deleted)
> > > #
> > > # Does not change the timestamps or touch in any way the source code, only .o files
> > > #
> > > function gitcheckout {
> > > new_branch=$1
> > > current_branch=`command git rev-parse --abbrev-ref HEAD`
> > >
> > > if [ -f ${new_branch} ]; then
> > > command git checkout ${new_branch}
> > > return 0
> > > fi
> > >
> > > if [ "${new_branch}" == "-b" ]; then
> > > command git rev-parse --verify $2 > /dev/null 2>&1
> > > if [ "$?" == "0" ]; then
> > > echo "Branch you are trying to create $2 already exists, aborting"
> > > return 9
> > > else
> > > # create the new branch, do not yet check it out
> > > new_branch=$2
> > > command git branch ${new_branch}
> > > fi
> > > else
> > > if [ "${current_branch}" == "${new_branch}" ]; then
> > > echo "Already in branch ${new_branch}"
> > > return 0
> > > fi
> > > changes=`command git status --untracked-files=no --porcelain`
> > > if [ "${changes}" != "" ]; then
> > > echo "You have modified or new files; not changing branches"
> > > echo ${changes}
> > > return 5
> > > fi
> > > fi
> > >
> > > current_arch=`branchtoarch ${current_branch}`
> > > new_arch=`branchtoarch ${new_branch}`
> > >
> > > isbranch=0
> > > command git rev-parse --verify ${new_branch} > /dev/null 2>&1
> > > if [ "$?" == "0" ]; then
> > > isbranch=1
> > > else
> > > command git rev-parse --verify origin/${new_branch} > /dev/null 2>&1
> > > if [ "$?" == "0" ]; then
> > > isbranch=1
> > > fi
> > > fi
> > > if [ "${isbranch}" == "0" ]; then
> > > echo "Branch ${new_branch} does not exist"
> > > return 3
> > > fi
> > >
> > > echo "Checking out branch ${new_branch} and PETSC_ARCH ${new_arch}"
> > >
> > > if [ "${PETSC_ARCH}" != "${current_arch}" ]; then
> > > echo "Current PETSC_ARCH ${PETSC_ARCH} not equal to expected value ${current_arch} unable to removed out-dated object files"
> > > else
> > > # delete all out-dated .o files
> > > if [[ -f ${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables && -d ${PETSC_DIR}/${PETSC_ARCH}/obj ]]; then
> > > pushd ${PETSC_DIR}
> > > make --dry-run -f gmakefile | sed -n 's?[[:print:][:cntrl:]]*-o[[:blank:]]\([-[:alpha:]/]*\.o\)[[:print:][:cntrl:]]*?\1?gp' | xargs rm -f
> > > popd
> > > fi
> > > fi
> > >
> > > command git checkout ${new_branch}
> > > if [ "$?" != "0" ]; then
> > > echo "Unable to checkout branch ${new_branch}"
> > > return 4
> > > fi
> > >
> > > export PETSC_ARCH=${new_arch}
> > > # update the timestamps of all .o that remain for this PETSC_ARCH and the library
> > > if [[ -f ${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables && -d ${PETSC_DIR}/${PETSC_ARCH}/obj ]]; then
> > > find ${PETSC_DIR}/${PETSC_ARCH}/obj -type f -name *.o -exec touch -m {} +
> > > touch -m ${PETSC_DIR}/${PETSC_ARCH}/lib/libpetsc*
> > > # /usr/bin/dsymutil libpetsc.3.011.3.dylib
> > > fi
> > > }
> > >
> > > #
> > > # An alias for git checkout that manages resetting the PETSC_ARCH to the branch name and preserving compiled code associated with the branch checked out
> > > #
> > > # Oana suggested the idea to save waiting for code to recompile after changing branches and the use of touch to force code to not get recompiled unnecessarily
> > > #
> > > # Usage:
> > > # git checkout branchname
> > > #
> > > # Examples:
> > > # $ git checkout next
> > > # Checking out branch next and PETSC_ARCH arch-none
> > > # Switched to branch 'next'
> > > # ~/Src/petsc (next<) arch-next
> > > #
> > > # $ git checkout master
> > > # Checking out branch master and PETSC_ARCH arch-master
> > > # Switched to branch 'master'
> > > # ~/Src/petsc (master=) arch-master
> > > #
> > > # $ git checkout -b barry/mynewbranch
> > > # Switched to branch 'barry/mynewbranch'
> > > # ~/Src/petsc (barry/mynewbranch=) arch-mynewbranch
> > > #
> > > # Uses the bash function gitcheckout
> > > #
> > > function git {
> > > groot=`command git rev-parse --absolute-git-dir 2> /dev/null`
> > > if [[ "$1" == "checkout" && "$@" != *"--help"* && "${PETSC_DIR}" != "" && "${PETSC_DIR}/.git" == "${groot}" ]]; then
> > > shift 1
> > > if [ "$1" == "git" ]; then
> > > shift 2
> > > fi
> > > gitcheckout "$@"
> > > else
> > > command git "$@"
> > > fi
> > > }
> > >
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://www.cse.buffalo.edu/~knepley/
> >
> >
> >
>
>
>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
More information about the petsc-dev
mailing list