www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - "lit"-based test suite

reply Johan Engelen <j j.nl> writes:
Hi all,
   I've come to like the "lit"-based test suite a lot. It is much 
easier (for me at least) to run those tests than to run the DMD 
testsuite, especially on Windows.

I propose:
- Move the lit python script from ./tests/ir to ./tests.
- Exclude the "d2" folder from lit testing
- Move the LDC-specific tests from the DMD testsuite into a few 
other ./tests/abc directories and run them with lit instead

A big immediate benefit that I see is that it will be easier to 
add testcases and that testcases can be added in the same 
merge/commit as the implementation/fix for the testcase. (the 
submodule thing is a little annoying)
Also, I think the lit system has more flexibility for testing.

What do you think?

cheers,
   Johan
Jan 03
parent reply David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:
On 3 Jan 2016, at 19:36, Johan Engelen via digitalmars-d-ldc wrote:
 - Move the LDC-specific tests from the DMD testsuite into a few other 
 ./tests/abc directories and run them with lit instead
For the tests that are truly specific to LDC this certainly makes sense. As fas as the general D tests are concerned that just happened to cause an issue in the LDC glue code but not DMD, I think we should actually try to upstream them to the DMD testsuite, as other compilers will also benefit from the improved coverage. (Not sure whether the powers that be agree with that, though.) For new test cases, we could add them to the lit-based suite first and upstream them every so often. How suitable is the lit-based workflow for fail_compilation diagnostic checks? I'm not very familiar with it (yet). Another question is that of organization. Dumping everything (dmd testsuite runner, general CMake stuff, lit config, individual lit test case files) directly into tests/ might get a bit messy in the long run. — David
Jan 03
parent Johan Engelen <j j.nl> writes:
On Sunday, 3 January 2016 at 18:59:16 UTC, David Nadlinger wrote:
 On 3 Jan 2016, at 19:36, Johan Engelen via digitalmars-d-ldc 
 wrote:
 - Move the LDC-specific tests from the DMD testsuite into a 
 few other ./tests/abc directories and run them with lit instead
For the tests that are truly specific to LDC this certainly makes sense. As fas as the general D tests are concerned that just happened to cause an issue in the LDC glue code but not DMD, I think we should actually try to upstream them to the DMD testsuite, as other compilers will also benefit from the improved coverage. (Not sure whether the powers that be agree with that, though.) For new test cases, we could add them to the lit-based suite first and upstream them every so often.
Agreed.
 How suitable is the lit-based workflow for fail_compilation 
 diagnostic checks? I'm not very familiar with it (yet).
For non-zero return values, there is the "not" LLVM utility: not ldc2 compile_error.d returns zero if ldc2 errored, and returns non-zero if ldc2 succeeded. There are many tests like that in LLVM/Clang. So then I think you'd get something like: // RUN: not %ldc2 %s 2&>1 | FileCheck %s // Test error diagnostic: // CHECK: error: bla bla.
 Another question is that of organization. Dumping everything 
 (dmd testsuite runner, general CMake stuff, lit config, 
 individual lit test case files) directly into tests/ might get 
 a bit messy in the long run.
I want to pull lit config, lit runner, cmake stuff out of /tests/ir, and put it in /tests. Then all testcases should go into a subdirectory of /tests: /tests/codegen, /tests/profile, tests/runnable, /tests/whatever.
Jan 03