[petsc-dev] Thoughts on pushing current CI infrastructure to the next level

[Karl Rupp]

Dear PETSc developers,

the current Jenkins server went live last summer. Since then, the 
stability of master and next has indeed improved. Who would have thought 
three years ago that `next` is almost as stable as `master`?

However, over the weeks and months some weaknesses of our current 
continuous integration infrastructure became apparent:

1.) Still no Jenkins tests on Windows, because the remote execution of a 
Java application has some issues with Cygwin (which we require for PETSc).

2.) Jenkins workers every once in a while hang on the target machine 
(this has been independently observed in a different setting by Jed as 
well).

3.) Nonscalability of the current setup: The Jenkins server clones a 
separate copy of the repository for each pull request and each test 
arch. Each clone of the PETSc repository is 300 MB, so if we aim at 40 
different arches (i.e. the current coverage of the nightly tests) to 
test for each pull request, 300 MB * 40 = 12 GB of memory is required 
*for each pull request* on the Jenkins master.

4.) Pull requests from external repositories in Bitbucket are currently 
tested by Jenkins, but the results are not visible on the pull requests 
page. This might be a Bitbucket issue rather than a Jenkins issue; and 
yet, it impedes our work flow.

5.) Adding additional workers requires significant configuration effort 
on the Jenkins master and is far from hassle-free. For example, it is 
currently impractical to add my office machine to the pool of workers, 
even though this machine is 99% idle.

With some effort we can certainly address 1.) and to some extent 3.), 
probably 4.) as well, but I don't know how to solve 2.) and 5.) with 
Jenkins. Given that a significant effort is required for 1.), 3.) and 
4.) anyway, I'm starting to get more and more comfortable with the idea 
of rolling our own CI infrastructure (which has been suggested in some 
of Barry's snarky remarks already ;-) ). Small Python scripts for 
executing the tests and pushing results to Bitbucket as well as a 
central result storage can replicate our existing setup with a few lines 
of codes, while being much more flexible.

What do other PETSc developers think about CI infrastructure? Maybe 
suggestions other than Jenkins?

Best regards,
Karli