On Sun, Sep 16, 2012 at 10:46 PM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div class="im">On Sun, Sep 16, 2012 at 10:13 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
What are the various plans for running the test cases on examples?<br>
<br>
As I've said before I'd really like each example to be self-contained which means its tests and its test results would all sit in the same file (so, for example, the file could trivially be moved to another directory location and the tests still work). One way to do this is to have each example file to have a chunk of python code after the source that ran all the tests and then after that a chunk of all the outputs. Having the test cases inside the makefile (or in some directory or gasp "single location specific file) and the output files in the output directory is, to me, not desirable.<br>
</blockquote><div><br></div></div><div>It sounds like you are asking for something like Python's doctest for executables.</div><div><br></div><div><a href="http://docs.python.org/library/doctest.html" target="_blank">http://docs.python.org/library/doctest.html</a></div>
<div><br></div><div>If the output was feasibly short, we could build a system this way.</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Jed, what is your horrible plan?<br></blockquote><div><br></div></div><div>I wrote a parser for our makefile-based tests. It currently creates files that describe tests like this:</div><div><br></div><div><div>with Executable('ex15.c', libs='PETSC_TS_LIB'):</div>
<div> Test(id='1', args='-da_grid_x 20 -da_grid_y 20 -boundary 0 -ts_max_steps 10')</div><div> Test(id='2', args='-da_grid_x 20 -da_grid_y 20 -boundary 0 -ts_max_steps 10 -Jtype 2', compare='ex15_1')</div>
<div> Test(id='3', args='-da_grid_x 20 -da_grid_y 20 -boundary 1 -ts_max_steps 10')</div><div> Test(id='4', np=2, args='-da_grid_x 20 -da_grid_y 20 -boundary 1 -ts_max_steps 10')</div>
<div> Test(id='5', args='-da_grid_x 20 -da_grid_y 20 -boundary 0 -ts_max_steps 10 -Jtype 1', compare='ex15_1')</div></div><div><br></div><div>I would the "id" field to use more descriptive names when appropriate so these numbers are just the inherited names. This snippet of Python registers the executable and the associated tests in a global index (will probably make at a sqlite database). The test result can be reported back to the database. I wrote most of a parallel executor with active progress monitoring, similar to nose-progressive, but not all the pieces are playing nicely together yet.</div>
<div><br></div><div>Me eventual plan is to be able to batch up the test results, which will include basic timing information, and report it back to a central server so we can keep a log. That could be done with buildbot or Jenkins, which fulfill most of our needs for a "dashboard", but lack a good database query API.</div>
<div><br></div><div>Test specification, which seems to be what you are most concerned with, could also be via "doctests". The main thing is that if you execute the test script, it first updates its database with all executables and tests in subdirectories (specified any way we like), then does whatever operations you ask for (building, testing, etc).</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Matt, what is your horrible plan?<br></blockquote></div></div></blockquote><div><br></div><div>What I currently have:</div><div><br></div><div> Very much like Jed, except I use a dictionary instead of a Test() object, which in Python is almost the same thing.</div>
<div>Could be converted in 10min.</div><div><br></div><div>What I want:</div><div><br></div><div> I have no problem with regression, but I do not think it makes a difference whether that output is in a separate file.</div>
<div>What I really want is unit-style tests for things. For instance, I want norms for iterates as objects, so I can test</div><div>them with a certain relative tolerance. I want solver descriptions as objects so I can just test for equality, etc.</div>
<div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Can we converge to something agreeable to all?<br>
<span><font color="#888888"><br>
Barry<br>
<br>
<br>
</font></span></blockquote></div></div><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>