[petsc-dev] plans for testing examples?

[Matthew Knepley]

On Sun, Sep 16, 2012 at 10:46 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Sun, Sep 16, 2012 at 10:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>    What are the various plans for running the test cases on examples?
>>
>>     As I've said before I'd really like each example to be self-contained
>> which means its tests and its test results would all sit in the same file
>> (so, for example, the file could trivially be moved to another directory
>> location and the tests still work).  One way to do this is to have each
>> example file to have a chunk of python code after the source that ran all
>> the tests and then after that a chunk of all the outputs.   Having the test
>> cases inside the makefile (or in some directory or gasp "single location
>> specific file) and the output files in the output directory is, to me, not
>> desirable.
>>
>
> It sounds like you are asking for something like Python's doctest for
> executables.
>
> http://docs.python.org/library/doctest.html
>
> If the output was feasibly short, we could build a system this way.
>
>
>>
>>    Jed,  what is your horrible plan?
>>
>
> I wrote a parser for our makefile-based tests. It currently creates files
> that describe tests like this:
>
> with Executable('ex15.c', libs='PETSC_TS_LIB'):
>     Test(id='1', args='-da_grid_x 20 -da_grid_y 20 -boundary 0
> -ts_max_steps 10')
>     Test(id='2', args='-da_grid_x 20 -da_grid_y 20 -boundary 0
> -ts_max_steps 10 -Jtype 2', compare='ex15_1')
>     Test(id='3', args='-da_grid_x 20 -da_grid_y 20 -boundary 1
> -ts_max_steps 10')
>     Test(id='4', np=2, args='-da_grid_x 20 -da_grid_y 20 -boundary 1
> -ts_max_steps 10')
>     Test(id='5', args='-da_grid_x 20 -da_grid_y 20 -boundary 0
> -ts_max_steps 10 -Jtype 1', compare='ex15_1')
>
> I would the "id" field to use more descriptive names when appropriate so
> these numbers are just the inherited names. This snippet of Python
> registers the executable and the associated tests in a global index (will
> probably make at a sqlite database). The test result can be reported back
> to the database. I wrote most of a parallel executor with active progress
> monitoring, similar to nose-progressive, but not all the pieces are playing
> nicely together yet.
>
> Me eventual plan is to be able to batch up the test results, which will
> include basic timing information, and report it back to a central server so
> we can keep a log. That could be done with buildbot or Jenkins, which
> fulfill most of our needs for a "dashboard", but lack a good database query
> API.
>
> Test specification, which seems to be what you are most concerned with,
> could also be via "doctests". The main thing is that if you execute the
> test script, it first updates its database with all executables and tests
> in subdirectories (specified any way we like), then does whatever
> operations you ask for (building, testing, etc).
>
>
>>
>>    Matt, what is your horrible plan?
>>
>
What I currently have:

  Very much like Jed, except I use a dictionary instead of a Test() object,
which in Python is almost the same thing.
Could be converted in 10min.

What I want:

  I have no problem with regression, but I do not think it makes a
difference whether that output is in a separate file.
What I really want is unit-style tests for things. For instance, I want
norms for iterates as objects, so I can test
them with a certain relative tolerance. I want solver descriptions as
objects so I can just test for equality, etc.

    Matt


>     Can we converge to something agreeable to all?
>>
>>    Barry
>>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120917/cb3e92bc/attachment.html>