[petsc-dev] plans for testing examples?

[Barry Smith]

On Sep 17, 2012, at 10:12 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Mon, Sep 17, 2012 at 9:44 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> One reason I like everything in one file/place is that now lots of "test" examples are in the "tutorial" directories. To move them requires:
> 
> 1)  hg move the example code
> 2) hg move several output/* files
> 3) edit the "tutorials" makefile to remove stuff
> 4) edit the "test" makefile to add the stuff
> 
>   and plus since there will be name conflicts for source code names one will also need to change, for example, ex1.c to ex37.c in all these places.
> 
> I think the dumb numbers were a bad idea in the first place. There are way too many examples that are ("this is a less maintained copy of some other Bratu example"). How are we going to update inline output? We could write an Emacs minor mode that updated it on file save, but to keep file save fast enough, the output would have to be lying around. We'd also have to be careful not to accidentally check in bad output. I hate the idea of having to manually copy and paste into some piece of the output.
>  
>   If everything was in one file I could just hg move that one file to the new location (changing the name of the file if needed). Sooo much easier. Why do you think no one has fixed the cluster fuck we have currently.
> 
> What about storing a SHA1 of the reference output in the source file (or wherever the test was specified).

   Almost there. We need a unique way of mapping from each test to a unique key that can be used to look in your big old output database.  It would be best if this key came automatically, unfortunately the command line options are not unique to a run and if we did a hash on the source code that would change with every trivial change to the source code.  I don't want to use the filename and or directory name because that makes moving examples around more painful.  Hash the help string AND the command line options together is that close enough to unique?

   Note also: in the same place in the example code we provide the "chunk of python for registering tests" we can provide a "chunk of python to register information about the example": what dimension, what equation, what type of solver (TS, SNES, KSP) what type of DM etc thus generate automatically a decent searchable database of examples.  By using "a chunk of python" rather than defining some "doctext" language we can avoid any need to parse.


   Barry




> Then have one big store of outputs for everything, indexed by SHA1. The output store need not even be part of the Hg checkout, it could be stored in a separate repository. When reference output is updated, a new file is created in the archive and the SHA1 at the test specification is updated. We could make a more convenient diff tool, but this allows moving files around willy-nilly, including taking them entirely outside of PETSc, without needing to futz with the test output files.
> 
> If we did something like this, I'd be more comfortable specifying tests with markup inside the source file (though I'd rather that not be the _only_ way to specify tests).