www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Let's improve the dmd tester.

reply Stefan Koch <uplink.coder googlemail.com> writes:
Lately,

I have been working quite a lot on DMD again.
(Thanks to my Employer who is using my experience).

One thing which makes intrusive changes to DMD, hard to do, is 
the testing "script".
Which executes and evaluates tests for DMD.

It greets you with a wall of text, in which passing and failing 
tests are happily coexisting.
Which means that seeing which test failed for example on the 
output of Ci service, in which you can't do a grep, is a 
multi-second barrier in ones development process.

I do believe that fixing that, will have a large positive impact 
on the stability and maintainability of dmd.
Which is important since it's still the main semantic engine 
behind D.

I am going to try and fix the tester, but help and suggestions 
are always welcome.

Greetings,

Stefan
Jun 25
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jun 25, 2020 at 02:39:18PM +0000, Stefan Koch via Digitalmars-d wrote:
[...]
 Which means that seeing which test failed for example on the output of
 Ci service, in which you can't do a grep, is a multi-second barrier in
 ones development process.
[...] Seriously, why isn't there a CI service out there that allows you instant access to the text output *without* requiring you to use a browser on some over-complex, over-engineered JS-encumbered website? I mean, c'mon people, it's *plaintext* for crying out loud. We're not talking about 3D-accelerated FPS-in-a-browser here. Isn't there some URL that you can just curl to access the output instantly? T -- A mathematician learns more and more about less and less, until he knows everything about nothing; whereas a philospher learns less and less about more and more, until he knows nothing about everything.
Jun 25
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/25/20 12:29 PM, H. S. Teoh wrote:
 On Thu, Jun 25, 2020 at 02:39:18PM +0000, Stefan Koch via Digitalmars-d wrote:
 [...]
 Which means that seeing which test failed for example on the output of
 Ci service, in which you can't do a grep, is a multi-second barrier in
 ones development process.
[...] Seriously, why isn't there a CI service out there that allows you instant access to the text output *without* requiring you to use a browser on some over-complex, over-engineered JS-encumbered website? I mean, c'mon people, it's *plaintext* for crying out loud. We're not talking about 3D-accelerated FPS-in-a-browser here. Isn't there some URL that you can just curl to access the output instantly?
Not to mention there seems to be a contest among CI engines on which manages to make the error lines in a build more difficult to find.
Jun 25
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 25 June 2020 at 16:29:24 UTC, H. S. Teoh wrote:
 On Thu, Jun 25, 2020 at 02:39:18PM +0000, Stefan Koch via 
 Digitalmars-d wrote: [...]
 Which means that seeing which test failed for example on the 
 output of Ci service, in which you can't do a grep, is a 
 multi-second barrier in ones development process.
[...] Seriously, why isn't there a CI service out there that allows you instant access to the text output *without* requiring you to use a browser on some over-complex, over-engineered JS-encumbered website? I mean, c'mon people, it's *plaintext* for crying out loud. We're not talking about 3D-accelerated FPS-in-a-browser here. Isn't there some URL that you can just curl to access the output instantly? T
Sounds like an entrepreneurial opportunity to me.
Jun 25
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2020-06-25 16:39, Stefan Koch wrote:
 Lately,
 
 I have been working quite a lot on DMD again.
 (Thanks to my Employer who is using my experience).
 
 One thing which makes intrusive changes to DMD, hard to do, is the 
 testing "script".
 Which executes and evaluates tests for DMD.
 
 It greets you with a wall of text, in which passing and failing tests 
 are happily coexisting.
 Which means that seeing which test failed for example on the output of 
 Ci service, in which you can't do a grep, is a multi-second barrier in 
 ones development process.
 
 I do believe that fixing that, will have a large positive impact on the 
 stability and maintainability of dmd.
 Which is important since it's still the main semantic engine behind D.
 
 I am going to try and fix the tester, but help and suggestions are 
 always welcome.
The tests which are located in test/unit have a separate test runner (invoked by the main one) which only prints a summary when all tests are successful. You can invoke them explicitly using `test/run.d -u`: $ ./test/run.d -u unit_test_runner is already up-to-date 24 tests, 0 failures It has several advantages to the other test runner: * Compiles all tests into one executable, this will result in faster execution times * Since all tests are compiled into one executable it's much more scalable then the main test runner. One can freely create directories and files to organize the tests without slowing it down too much * Since the tests and the compiler runs in the same executable the tests have access to the compiler internals, make it much easier to test certain parts. For example, a test for the lexer does only need to invoke the lexer, not the whole compiler including code generation * All tests have a UDA which describes what each test does * It's possible to filter tests by passing only the files you would like to test: $ ./test/run.d -u test/unit/deinitialization.d unit_test_runner is already up-to-date 7 tests, 0 failures * It's possible to filter tests by UDA: $ ./test/run.d -u --filter Expression.deinitialize unit_test_runner is already up-to-date 1 tests, 0 failures * When a test fails it prints a nice output, including stacktrace, the UDAs attach to the tests and a summary of the failing tests: $ ./test/run.d -u test/unit/deinitialization.d unit_test_runner is already up-to-date Failures: 1) global.deinitialize core.exception.AssertError /Users/doob/development/d/dlang/dmd/test/unit/dei itialization.d(28): unittest failure ---------------- ??:? _d_unittestp [0x10f0687a1] /Users/doob/development/d/dlang/dmd/test/unit/deinitialization.d:28 void deinitialization.__unittest_L4_C1() [0x10edf4380] /Users/doob/development/d/dlang/dmd/test/unit/deinitialization.d:3 core.runtime.UnitTestResult runner.unitTestRunner() [0x10ee02b57] ??:? runModuleUnitTests [0x10f069182] ??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).runAll() [0x10f08436c] ??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).tryExec(scope void delegate()) [0x10f0842f8] ??:? _d_run_main2 [0x10f08425c] ??:? _d_run_main [0x10f083fd9] __main.d:1 main [0x10edf1e1d] ??:? start [0x7fff5f1473d4] 7 tests, 1 failures Failed tests: /Users/doob/development/d/dlang/dmd/test/unit/deinitialization.d:28 I recommend all tests in "tests/compilable" and "tests/fail_compilation" to be converted to the style of tests in "tests/unit". -- /Jacob Carlborg
Jun 25
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 25 June 2020 at 14:39:18 UTC, Stefan Koch wrote:
 One thing which makes intrusive changes to DMD, hard to do, is 
 the testing "script".
 Which executes and evaluates tests for DMD.
You do realize that you can run all tests locally? For example, `./test/run.d compilable` takes less than ten seconds on my machine. You can also execute only individual tests e.g. `./test/run.d compilable/dtoh_enum.d`
 It greets you with a wall of text, in which passing and failing 
 tests are happily coexisting.
The tester ensures that the log output of each test isn't mixed.
 Which means that seeing which test failed for example on the 
 output of Ci service, in which you can't do a grep, is a 
 multi-second barrier in ones development process.
So to summarize your "wishes": a) print error messages of failed tests at the end of the execution BTW for local testing you can just rerun the test script. It will only rerun the failing tests. b) hide messages for successful tests under the verbose flag Did I understand you correctly?
Jun 25
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 25 June 2020 at 18:15:57 UTC, Seb wrote:
 On Thursday, 25 June 2020 at 14:39:18 UTC, Stefan Koch wrote:
 [...]
You do realize that you can run all tests locally? For example, `./test/run.d compilable` takes less than ten seconds on my machine. You can also execute only individual tests e.g. `./test/run.d compilable/dtoh_enum.d`
 [...]
The tester ensures that the log output of each test isn't mixed.
 [...]
So to summarize your "wishes": a) print error messages of failed tests at the end of the execution BTW for local testing you can just rerun the test script. It will only rerun the failing tests. b) hide messages for successful tests under the verbose flag Did I understand you correctly?
Yes that would be the first things which should be fixed. The point is that I cannot find everything by running locally, for example I don't have a mac. And I don't use windows for development.
Jun 25
prev sibling next sibling parent Nils Lankila <NilsLankila gmx.us> writes:
On Thursday, 25 June 2020 at 18:15:57 UTC, Seb wrote:
 On Thursday, 25 June 2020 at 14:39:18 UTC, Stefan Koch wrote:
 [...]
You do realize that you can run all tests locally? For example, `./test/run.d compilable` takes less than ten seconds on my machine. You can also execute only individual tests e.g. `./test/run.d compilable/dtoh_enum.d`
 [...]
The tester ensures that the log output of each test isn't mixed.
 [...]
So to summarize your "wishes": a) print error messages of failed tests at the end of the execution BTW for local testing you can just rerun the test script. It will only rerun the failing tests.
That's generally not true as between two runs you change the compiler. Failed scripts are rerun when you fix a test, for example the TEST_OUTPUT section, which happens but less frequently,
 b) hide messages for successful tests under the verbose flag

 Did I understand you correctly?
Jun 25
prev sibling next sibling parent Nils Lankila <NilsLankila gmx.us> writes:
On Thursday, 25 June 2020 at 18:15:57 UTC, Seb wrote:
 On Thursday, 25 June 2020 at 14:39:18 UTC, Stefan Koch wrote:
 One thing which makes intrusive changes to DMD, hard to do, is 
 the testing "script".
 Which executes and evaluates tests for DMD.
You do realize that you can run all tests locally? For example, `./test/run.d compilable` takes less than ten seconds on my machine. You can also execute only individual tests e.g. `./test/run.d compilable/dtoh_enum.d`
 It greets you with a wall of text, in which passing and 
 failing tests are happily coexisting.
The tester ensures that the log output of each test isn't mixed.
 Which means that seeing which test failed for example on the 
 output of Ci service, in which you can't do a grep, is a 
 multi-second barrier in ones development process.
So to summarize your "wishes": a) print error messages of failed tests at the end of the execution BTW for local testing you can just rerun the test script. It will only rerun the failing tests. b) hide messages for successful tests under the verbose flag
A status line (same line updated, no new line) would be nice... in the form of %s test(s) run, %d test(s) failed, %d test(s) passed and then the summary of failed tests at the end, to prevent the problem of interleaving.
Jun 25
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/25/2020 11:15 AM, Seb wrote:
 On Thursday, 25 June 2020 at 14:39:18 UTC, Stefan Koch wrote:
 One thing which makes intrusive changes to DMD, hard to do, is the testing 
 "script".
 Which executes and evaluates tests for DMD.
You do realize that you can run all tests locally? For example, `./test/run.d compilable` takes less than ten seconds on my machine. You can also execute only individual tests e.g. `./test/run.d compilable/dtoh_enum.d`
One of my issues with this is: 1. how is the compiler version that is used to build test/run.d determined? 2. how is the compiler version that runs the test(s) determined? 3. how is the import path to druntime/phobos set for (1) and (2) ? Please don't answer here, add a PR to fix run.d and the corresponding README.md. https://issues.dlang.org/show_bug.cgi?id=20979 I've added the keyword "TestSuite" for test suite issues: https://issues.dlang.org/buglist.cgi?keywords=TestSuite&list_id=232009 Please add all test suite issues under that keyword. And my previous grumpy rant: https://digitalmars.com/d/archives/digitalmars/D/Serious_Problems_with_the_Test_Suite_339807.html
Jun 25
parent Seb <seb wilzba.ch> writes:
On Friday, 26 June 2020 at 01:04:50 UTC, Walter Bright wrote:
 One of my issues with this is:

 1. how is the compiler version that is used to build test/run.d 
 determined?
That's entirely up to `rdmd`. It picks the compiler binary in your path, i.e. the host compiler you used to build dmd.
 2. how is the compiler version that runs the test(s) determined?
There's _NO_ version determination. It uses the newly generated dmd binary that was freshly generated, i.e. it only looks into the `generated` folder. Details: https://github.com/dlang/dmd/blob/master/test/tools/paths.d
 3. how is the import path to druntime/phobos set for (1) and 
 (2) ?
As for all compilation in D: by DMD's configuration file (dmd.ini for Posix or sc.ini on Windows). build.d ensures that a proper config files get generated. DMD's configuration lookup sequence is far from ideal, but I gave up on improving it (see e.g. [1]) as I gave up on DMD for anything substantial. [1] https://github.com/dlang/dmd/pull/7915
 Please don't answer here, add a PR to fix run.d and the 
 corresponding README.md.
Well, then I better not answer.
Jun 26