<div class="gmail_quote">On Tue, Feb 21, 2012 at 12:37 PM, David Cole <span dir="ltr"><<a href="mailto:david.cole@kitware.com">david.cole@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div><div class="h5">On Tue, Feb 21, 2012 at 1:27 PM, Robert Dailey <span dir="ltr"><<a href="mailto:rcdailey@gmail.com" target="_blank">rcdailey@gmail.com</a>></span> wrote:<br></div></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi,<div><br></div><div>I'm using Visual Studio as my generator for my CMake projects. As of right now, I make my tests depend on the libraries they test. So for example, tests named:</div><div><br></div><div>test_thingA</div>
<div>test_thingB</div><div><br></div><div>will all depend on library:</div><div><br></div><div>libfoo.lib</div><div><br></div><div>When I build target "libfoo" in visual studio, it would be nice to have all dependent tests build as well, and have them each execute.</div>
<div><br></div><div>The goal for all of this is to make it as convenient as possible for developers on my team to RUN TESTS on their code before they submit to version control. I want to make it automated, so when they rebuild the library, the testing automatically happens. I'd also obviously create an option in cmake cache to turn this automation off should it become too annoying.</div>
<div><br></div><div>If this isn't a good idea, can someone recommend a good workflow for running tests locally prior to checking in source code?<br clear="all"><div><br></div><div>---------</div>Robert Dailey<br>
</div>
<br></div></div>--<br>
<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the CMake FAQ at: <a href="http://www.cmake.org/Wiki/CMake_FAQ" target="_blank">http://www.cmake.org/Wiki/CMake_FAQ</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://www.cmake.org/mailman/listinfo/cmake" target="_blank">http://www.cmake.org/mailman/listinfo/cmake</a><br></blockquote></div><br><div><br></div><div>If you're using add_test in your CMakeLists files, then the perfect way to prove that the tests all work on a developer's machine is for him or her to run:</div>
<div><br></div><div> ctest -D Experimental</div><div><br></div><div>after making local mods, and before pushing the changes to your source control system.</div><div><br></div><div>That will configure, build all and run all the tests. And submit the results to your CDash server so that is public evidence that he actually did run the tests, and hopefully that they all passed on his machine at least.</div>
<div><br></div><div>You can also restrict the set of tests that run using -R or -I or -L on the ctest command line, although, you should strive to have your test suite be brief enough that it's not painful for folks to run the full test suite prior to checkin.</div>
</blockquote><div><br></div><div>I think this is a reasonable idea for small projects, but in general I disagree with running all tests.</div><div><br></div><div>There are hundreds of projects (probably 150) and hundreds more of tests (probably 10 tests per project). In general agile methodology, it only makes sense to unit test those components which have changed. Unit testing a dependent component that did not have a source code change will not be needed or beneficial.</div>
<div><br></div><div>All of these tests can take hours to run, which isn't unacceptable because it's a full test suite. Only the build server kicks off a build and runs the FULL test suite (thus running ctest -D Experimental as you have suggested). Developers just do an intermediate check by unit testing only the parts of the code base that have changed. This is essential for practices like continuous integration.</div>
<div><br></div><div>Ideally the pipeline goes like this:</div><div><ol><li>Programmer makes a change to a certain number of libraries</li><li>Programmer runs the relevant tests (or all) for each of the libraries that were changed.</li>
<li>Once those tests have passed, the developer submits the source code to version control</li><li>The build server is then instructed to run a full build and test of the entire code base for each checkin.</li><li>The build server can then run any integration tests that are configured (not sure how these would be setup in CMake - probably again as tests, but not specific to only a single project)</li>
<li>Build is considered "complete" at this point.</li></ol><div>Seems like there would be no choice but to run them individually in this case, since CMake really shines only in steps after #3</div></div></div>