www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How can one reliably run unittests

reply deadalnix <deadalnix gmail.com> writes:
D's unittest block is a great idea. However, I find that in 
practice, I fail to make good use of them and have been 
neglecting them. Sometime by using an external test suite 
instead, sometime neglecting testing altogether. How come?

Well, the main reason is that, as far as I can tell, it is near 
damn impossible to run them at scale in any sensible way. Let me 
explain.

Ideally, one would want a module to be a unit. Declaration, 
implementation, but also tests are part of the unit. During the 
dev cycle, it is desired to be able to runt he test on modules to 
verify that everything works as advertised. While doing so is 
easy on a pet project,a s the projects grows, contains several 
libraries and executable, doing this become very challenging.

It is not really possible to build a monster executable that 
contains everything. First, this would kill build times, but, 
with several main around it won't link. So let's split into 
component.

Some of these component are libraries, some are executable. 
Adding a main for libraries is required, or it won't link, but 
adding one to executable is going to cause a link error. This in 
itself is a major main in the ass, because that means there is no 
one consistent way to unittest a module without knowing if that 
module has a main. In addition, everything needs to be built 
twice now, which is really undesirable.

Maybe one could rdmd each modules to runt he tests? That seems 
like the best approach to me, considering one doesn't want to 
actually produce an artifact for the tests, simply run them. this 
also has the main/no main problem, but in addition, it often 
fails with incomprehensible linker errors. For some reason, rdmd 
is not able to include all dependencies consistently, but, even 
better, it doesn't take the same standard flags as other linkers 
do, so it is not possible to reuse existing build infrastructure 
to feed all the correct flags to rdmd.

This may seems like it wouldn't be that big of a deal if you 
manage all your flags by yourself, but very quickly turns into a 
nightmare once you have to use 3rd party libraries that ship with 
their own set of flags.

At this point, I would just wish that one could rdmd --unittest 
modulename.d and just pass it a couple of -I, -L and -l flags and 
have all of it work. I have no idea how to achieve that.

This kind of orthogonality is important as a project scale. One 
cannot just doctor all the unitests call to all the modules. You 
need to be able to tell your build system something akin to "hey, 
every time you encounter a D module, please implicitly add this 
unitest target. Thanks." This cannot be made to work if running 
unit tests is done in a way that depends on the content of the 
module.

I don't think my case is isolated, every single substantial D 
project I encountered has some kind of test suite or something 
similar instead of relying on the unitests within modules. This 
is frankly a failure, and it is 100% due to poor UX as the 
language feature itself is great.

If someone has some bandwidth, it is IMO a high leverage task 
that solve real problem for real people and 90% of the work is 
already there. The feature is there, the runtime support &al is 
there. just provide a consistent UX so its use can be integrated 
properly in larger systems.

Thanks in advance.
Aug 24
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 24 August 2021 at 12:21:41 UTC, deadalnix wrote:
 D's unittest block is a great idea. However, I find that in 
 practice, I fail to make good use of them and have been 
 neglecting them. Sometime by using an external test suite 
 instead, sometime neglecting testing altogether. How come?

 Well, the main reason is that, as far as I can tell, it is near 
 damn impossible to run them at scale in any sensible way. Let 
 me explain.

 Ideally, one would want a module to be a unit. Declaration, 
 implementation, but also tests are part of the unit. During the 
 dev cycle, it is desired to be able to runt he test on modules 
 to verify that everything works as advertised. While doing so 
 is easy on a pet project,a s the projects grows, contains 
 several libraries and executable, doing this become very 
 challenging.

 It is not really possible to build a monster executable that 
 contains everything. First, this would kill build times, but, 
 with several main around it won't link. So let's split into 
 component.

 Some of these component are libraries, some are executable. 
 Adding a main for libraries is required, or it won't link, but 
 adding one to executable is going to cause a link error. This 
 in itself is a major main in the ass, because that means there 
 is no one consistent way to unittest a module without knowing 
 if that module has a main. In addition, everything needs to be 
 built twice now, which is really undesirable.

 Maybe one could rdmd each modules to runt he tests? That seems 
 like the best approach to me, considering one doesn't want to 
 actually produce an artifact for the tests, simply run them. 
 this also has the main/no main problem, but in addition, it 
 often fails with incomprehensible linker errors. For some 
 reason, rdmd is not able to include all dependencies 
 consistently, but, even better, it doesn't take the same 
 standard flags as other linkers do, so it is not possible to 
 reuse existing build infrastructure to feed all the correct 
 flags to rdmd.

 This may seems like it wouldn't be that big of a deal if you 
 manage all your flags by yourself, but very quickly turns into 
 a nightmare once you have to use 3rd party libraries that ship 
 with their own set of flags.

 At this point, I would just wish that one could rdmd --unittest 
 modulename.d and just pass it a couple of -I, -L and -l flags 
 and have all of it work. I have no idea how to achieve that.

 This kind of orthogonality is important as a project scale. One 
 cannot just doctor all the unitests call to all the modules. 
 You need to be able to tell your build system something akin to 
 "hey, every time you encounter a D module, please implicitly 
 add this unitest target. Thanks." This cannot be made to work 
 if running unit tests is done in a way that depends on the 
 content of the module.

 I don't think my case is isolated, every single substantial D 
 project I encountered has some kind of test suite or something 
 similar instead of relying on the unitests within modules. This 
 is frankly a failure, and it is 100% due to poor UX as the 
 language feature itself is great.

 If someone has some bandwidth, it is IMO a high leverage task 
 that solve real problem for real people and 90% of the work is 
 already there. The feature is there, the runtime support &al is 
 there. just provide a consistent UX so its use can be 
 integrated properly in larger systems.

 Thanks in advance.
Interesting. My ideal scenario wouldn't involve linkers at all - I don't want to spend time waiting for my code to link, especially since, the way I write code means that unit tests never need optimising or native code generated. I'd go as far as to say that if one's unit tests *do* need any of that, they need to rethink their mini-integra... err, unit tests. But I digress. The way I'd like it to work personally would be to CTFE unit tests in a module and have them run whenever I save the file. No linker, no main function, just that. Back to the issue at hand. The problem with wanting to pass flags to the rdmd is that unless the project is trivial it'll have dub dependencies, so doing that manually is a no-no. This is the same thing that happens all the time in compile-as-you-type plugins, and why I maintain two separate emacs packages to make that done automatically (one for CMake/C++ and one for dub/D). Which means asking dub. Which means a build system. Which means you just gave me an idea for an awesome reggae rule to generate a library with all of the dependencies of a module, a main function to run the tests, and then voilà. It'd have to do this for each module, and the first time said all-dependencies library is built it'd take time, but after that it'd work. You'd still have to wait for the linker though, and if for some reason you're not using lld (or worse, you're on Windows), the wait will be looooong.
Aug 24
next sibling parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 24 August 2021 at 12:39:47 UTC, Atila Neves wrote:
 My ideal scenario wouldn't involve linkers at all - I don't 
 want to spend time waiting for my code to link, especially 
 since, the way I write code means that unit tests never need 
 optimising or native code generated. I'd go as far as to say 
 that if one's unit tests *do* need any of that, they need to 
 rethink their mini-integra... err, unit tests. But I digress.
This is not doable in practice ATM. You use an exception from another module? you are screwed already, you need to link that other module in. This is a UX disaster, though.
 The way I'd like it to work personally would be to CTFE unit 
 tests in a module and have them run whenever I save the file. 
 No linker, no main function, just that.
Yes, or if there is a linker behind, I don't want to know about it. Slow link time can be a problem, but making it work at all, even with slow link time, would be a tremendous step forward.
 Back to the issue at hand. The problem with wanting to pass 
 flags to the rdmd is that unless the project is trivial it'll 
 have dub dependencies, so doing that manually is a no-no. This 
 is the same thing that happens all the time in 
 compile-as-you-type plugins, and why I maintain two separate 
 emacs packages to make that done automatically (one for 
 CMake/C++ and one for dub/D). Which means asking dub. Which 
 means a build system.
yes, dub makes this whole problem exponentially worse.
 Which means you just gave me an idea for an awesome reggae rule 
 to generate a library with all of the dependencies of a module, 
 a main function to run the tests, and then voilà. It'd have to 
 do this for each module, and the first time said 
 all-dependencies library is built it'd take time, but after 
 that it'd work.
Yes. One problem you'll run into is the `main` issue, where you need to use different flags. It is also unfortunate that building system have to pay the cost to paper over bad command line UI. This is bound to create poor integration in existing toolchains - which we see is the case already.
 You'd still have to wait for the linker though, and if for some 
 reason you're not using lld (or worse, you're on Windows), the 
 wait will be looooong.
The alternative is the feature not being usable at all, so I don't really care. I can throw hardware at that problem.
Aug 24
prev sibling next sibling parent Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Tuesday, 24 August 2021 at 12:39:47 UTC, Atila Neves wrote:
 Interesting.

 My ideal scenario wouldn't involve linkers at all - I don't 
 want to spend time waiting for my code to link, especially 
 since, the way I write code means that unit tests never need 
 optimising or native code generated. I'd go as far as to say 
 that if one's unit tests *do* need any of that, they need to 
 rethink their mini-integra... err, unit tests. But I digress.

 The way I'd like it to work personally would be to CTFE unit 
 tests in a module and have them run whenever I save the file. 
 No linker, no main function, just that.

 Back to the issue at hand. The problem with wanting to pass 
 flags to the rdmd is that unless the project is trivial it'll 
 have dub dependencies, so doing that manually is a no-no. This 
 is the same thing that happens all the time in 
 compile-as-you-type plugins, and why I maintain two separate 
 emacs packages to make that done automatically (one for 
 CMake/C++ and one for dub/D). Which means asking dub. Which 
 means a build system.

 Which means you just gave me an idea for an awesome reggae rule 
 to generate a library with all of the dependencies of a module, 
 a main function to run the tests, and then voilà. It'd have to 
 do this for each module, and the first time said 
 all-dependencies library is built it'd take time, but after 
 that it'd work.

 You'd still have to wait for the linker though, and if for some 
 reason you're not using lld (or worse, you're on Windows), the 
 wait will be looooong.
Imho, there should be a well known api for fetching unit tests at compile time and at runtime, then external or builtin unit test runner could use the api to properly select which unit tests to run. Second thing that compile time api should offer is the ability to force compiler ignore unit tests from modules or packages or annotations on unit test itself (it is also a nightmare to compile unit tests from a third party source library that takes ages to compile due to template bloat), or in general provide some kind of hooks into unittest config and running, similar to how junit does in java. Best regards, Alexandru.
Aug 24
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/24/2021 5:39 AM, Atila Neves wrote:
 The way I'd like it to work personally would be to CTFE unit tests in a module 
 and have them run whenever I save the file. No linker, no main function, just
that.
Interestingly, a lot of the tests for ImportC are in the form of CTFE and _Static_assert(). It works so well, I plan on using that method wherever I can.
Aug 25
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 24 August 2021 at 12:21:41 UTC, deadalnix wrote:
 Some of these component are libraries, some are executable. 
 Adding a main for libraries is required, or it won't link, but 
 adding one to executable is going to cause a link error. This 
 in itself is a major main in the ass, because that means there 
 is no one consistent way to unittest a module without knowing 
 if that module has a main. In addition, everything needs to be 
 built twice now, which is really undesirable.
That particular problem has a well known workaround -- prepend the `main` function with `version` statements to exclude it from `unittest` builds: ```D version (unittest) {} else void main () { ... } ``` Obviously that assumes you control all the codebases that define a `main`, and it doesn't solve the problem of building everything twice, but it should cover a lot of use-cases.
Aug 24
parent reply deadalnix <deadalnix gmail.com> writes:
On Tuesday, 24 August 2021 at 15:54:25 UTC, Joseph Rushton 
Wakeling wrote:
 That particular problem has a well known workaround -- prepend 
 the `main` function with `version` statements to exclude it 
 from `unittest` builds:

 ```D
 version (unittest) {} else
 void main ()
 {
     ...
 }
 ```

 Obviously that assumes you control all the codebases that 
 define a `main`, and it doesn't solve the problem of building 
 everything twice, but it should cover a lot of use-cases.
True, and I'm sure that with an endless bag of trick, this could be made to work. this is missing the point, though. The compiler KNOWS if there is a main or not. The compiler KNOWS if I want unittests or not. Yet it's going to do retarded stuff unless one uses a bunch of workarounds. It's like we have this car, the engine works, the break are in order and so on, except the steering wheel is at the back seat and everybody tells you that you can still drive by passing your head through the window and look forward this way. This just doesn't make sense.
Aug 24
parent reply Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 24 August 2021 at 17:45:20 UTC, deadalnix wrote:
 The compiler KNOWS if there is a main or not. The compiler 
 KNOWS if I want unittests or not. Yet it's going to do retarded 
 stuff unless one uses a bunch of workarounds.
This. I think the original implementers had a different idea of how things would be used. For a long time, unittests would run first, then the main would be called. We all know how annoying that was, (most) people do not want this, they wanted two different modes, clearly separated. I say most, because there are still a few hold out I think. Personally, I'd like my unittest binary to generate something that has a much better UX than what's currently in druntime. You know, timing statistics, colors, ability to run a single test, the whole thing. I'm aware that there are frameworks out there, but it needs to be built-in.
Aug 25
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 25 August 2021 at 07:50:50 UTC, Mathias LANG wrote:
 This. I think the original implementers had a different idea of 
 how things would be used. For a long time, unittests would run 
 first, then the main would be called. We all know how annoying 
 that was, (most) people do not want this, they wanted two 
 different modes, clearly separated. I say most, because there 
 are still a few hold out I think.
I think there's also a complexity that the compiler cannot know which main to ignore (e.g. application main) versus which main to keep (e.g. a custom-written main inserted for running unittests, as some frameworks do). That's surely readily solvable but it suggests that there should be a ready way to distinguish between entry points for running an app versus entry points for running tests (so that test-suite implementations can override the latter and have it Just Work). I fully agree with you and Amaury that this _should_ be solved and that the user shouldn't have to bother with workarounds like the one that I shared above.
Aug 25
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/25/21 3:50 AM, Mathias LANG wrote:
 Personally, I'd like my unittest binary to generate something that has a 
 much better UX than what's currently in druntime. You know, timing 
 statistics, colors, ability to run a single test, the whole thing.
 
You can get almost all that via registering a [unittest handler function](https://dlang.org/phobos/core_runtime.html#.Runtime.extend dModuleUnitTester). The only thing that isn't provided is running individual tests, but that is a compiler issue (the compiler combines all unittests in a module into one callable).
 I'm aware that there are frameworks out there, but it needs to be built-in.
Disagree. I'm fine with the simple unittest experience, and avoiding putting all this kind of bloat into the runtime. -Steve
Aug 25
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven 
Schveighoffer wrote:
 The only thing that isn't provided is running individual tests, 
 but that is a compiler issue (the compiler combines all 
 unittests in a module into one callable).
You can run individual tests: ```d module tester1; unittest { assert(true); } unittest { assert(!!true); } unittest { assert(1 != 1); } unittest { assert(1 > 0); } version (unittest) { bool tester() { import std.meta : AliasSeq; import std.stdio : writef, writeln; alias tests = AliasSeq!(__traits(getUnitTests, tester1)); static foreach (i; 0 .. tests.length) { writef!"Test %d/%d ..."(i + 1, tests.length); try { tests[i](); writeln("ok"); } catch (Throwable t) { writeln("failed"); } } return false; } shared static this() { import core.runtime : Runtime; Runtime.moduleUnitTester = &tester; } } void main() { assert(false); // this doesn't get run } ``` usage: ``` $ dmd -unittest -run tester1.d Test 1/4 ...ok Test 2/4 ...ok Test 3/4 ...failed Test 4/4 ...ok ``` I found this out in https://forum.dlang.org/post/bukhjtbxouadyunqwdih forum.dlang.org , which has a tester that calls tests differently depending on their having a UDA
Aug 25
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/25/21 7:46 AM, jfondren wrote:
 On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven Schveighoffer wrote:
 The only thing that isn't provided is running individual tests, but 
 that is a compiler issue (the compiler combines all unittests in a 
 module into one callable).
You can run individual tests:
Well, yes. You can do everything yourself (you don't need to use the unittest system for this BTW). What I meant to say is, if you want to run all unit tests linked into the program, regardless of compile-time introspection, you need to rely on the compiler/linker to gather those for you (they are all provided as part of the ModuleInfo items), the compiler does not store references to individual tests. The things you *could* do are: 1. Print out messages for each module run, even if they pass 2. Colorize the output of those messages 3. Run unittests of specified modules only.
 I found this out in 
 https://forum.dlang.org/post/bukhjtbxouadyunqwdih forum.dlang.org , 
 which has a tester that calls tests differently depending on their 
 having a UDA
UDAs are also not provided to the ModuleInfo, so there isn't a way to affect the standard unittest framework that way. -Steve
Aug 25
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 25 August 2021 at 12:44:54 UTC, Steven 
Schveighoffer wrote:
 [snip]
Would it be possible to use an external unit testing framework to get all the tests when compiling with -betterC automatically without manually listing all the modules?
Aug 25
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/25/21 10:44 AM, jmh530 wrote:
 On Wednesday, 25 August 2021 at 12:44:54 UTC, Steven Schveighoffer wrote:
 [snip]
Would it be possible to use an external unit testing framework to get all the tests when compiling with -betterC automatically without manually listing all the modules?
betterC does not create ModuleInfo. So I don't think so. Potentially you could register tests via `pragma(crt_constructor)` calls. It would be messy. Nothing would be as nice as the way the compiler currently sets up the moduleinfo. -Steve
Aug 25
parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 25 August 2021 at 14:50:04 UTC, Steven 
Schveighoffer wrote:
 On 8/25/21 10:44 AM, jmh530 wrote:
 On Wednesday, 25 August 2021 at 12:44:54 UTC, Steven 
 Schveighoffer wrote:
 [snip]
Would it be possible to use an external unit testing framework to get all the tests when compiling with -betterC automatically without manually listing all the modules?
betterC does not create ModuleInfo. So I don't think so. Potentially you could register tests via `pragma(crt_constructor)` calls. It would be messy. Nothing would be as nice as the way the compiler currently sets up the moduleinfo. -Steve
Thanks.
Aug 25
prev sibling next sibling parent reply wjoe <invalid example.com> writes:
On Wednesday, 25 August 2021 at 11:46:18 UTC, jfondren wrote:
 On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven 
 Schveighoffer wrote:
 The only thing that isn't provided is running individual 
 tests, but that is a compiler issue (the compiler combines all 
 unittests in a module into one callable).
You can run individual tests: ```d module tester1; unittest { assert(true); } unittest { assert(!!true); } unittest { assert(1 != 1); } unittest { assert(1 > 0); } version (unittest) { bool tester() { import std.meta : AliasSeq; import std.stdio : writef, writeln; alias tests = AliasSeq!(__traits(getUnitTests, tester1)); static foreach (i; 0 .. tests.length) { writef!"Test %d/%d ..."(i + 1, tests.length); try { tests[i](); writeln("ok"); } catch (Throwable t) { writeln("failed"); } } return false; } shared static this() { import core.runtime : Runtime; Runtime.moduleUnitTester = &tester; } } void main() { assert(false); // this doesn't get run } ``` usage: ``` $ dmd -unittest -run tester1.d Test 1/4 ...ok Test 2/4 ...ok Test 3/4 ...failed Test 4/4 ...ok ``` I found this out in https://forum.dlang.org/post/bukhjtbxouadyunqwdih forum.dlang.org , which has a tester that calls tests differently depending on their having a UDA
But due to the fact that all unittests are compiled into one callable, i.e. it's all in the same process, doesn't that mean that after catching a throwable that the program is in UB ? And isn't that the reason why the runtime unittester aborts on the 1st failure ? So shouldn't it be like every module has its own callable which can then run in separate processes which then wouldn't affect others if one of them failed ? Also, that way all the modules could be tested in parallel.
Aug 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
 But due to the fact that all unittests are compiled into one 
 callable, i.e. it's all in the same process, doesn't that mean 
 that after catching a throwable that the program is in UB ? And 
 isn't that the reason why the runtime unittester aborts on the 
 1st failure ?
 So shouldn't it be like every module has its own callable which 
 can then run in separate processes which then wouldn't affect 
 others if one of them failed ? Also, that way all the modules 
 could be tested in parallel.
Errors thrown by assertions in unit tests are a special case:
 Individual tests are specified in the unit test using 
 AssertExpressions. Unlike AssertExpressions used elsewhere, the 
 assert is not assumed to hold, and upon assert failure the 
 program is still in a defined state.
Source: https://dlang.org/spec/unittest.html
Aug 25
next sibling parent reply wjoe <invalid example.com> writes:
On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
 On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
 [...]
Errors thrown by assertions in unit tests are a special case:
 [...]
Source: https://dlang.org/spec/unittest.html
That's good to know but I was specifically referring to the catch(Throwable) part in the above example.
Aug 25
parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 25 August 2021 at 13:28:40 UTC, wjoe wrote:
 On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
 On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
 [...]
Errors thrown by assertions in unit tests are a special case:
 [...]
Source: https://dlang.org/spec/unittest.html
That's good to know but I was specifically referring to the catch(Throwable) part in the above example.
Good point. It should probably be catch(AssertError), although even that is technically too broad.
Aug 25
prev sibling parent reply wjoe <invalid example.com> writes:
On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
 On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
 [...]
Errors thrown by assertions in unit tests are a special case:
 [...]
Source: https://dlang.org/spec/unittest.html
Also, does that hold for something like assertThrown!SomeError ?
Aug 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 25 August 2021 at 13:36:27 UTC, wjoe wrote:
 On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus wrote:
 On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
 [...]
Errors thrown by assertions in unit tests are a special case:
 [...]
Source: https://dlang.org/spec/unittest.html
Also, does that hold for something like assertThrown!SomeError ?
Officially, according to what's written in the spec? No. In practice? Yes. And enough people rely on custom assertions like assertThrown and the ones in unit-threaded [1] that this is unlikely to change. [1] https://unit-threaded.dpldocs.info/unit_threaded.assertions.html
Aug 25
parent wjoe <invalid example.com> writes:
On Wednesday, 25 August 2021 at 13:39:32 UTC, Paul Backus wrote:
 On Wednesday, 25 August 2021 at 13:36:27 UTC, wjoe wrote:
 On Wednesday, 25 August 2021 at 13:25:22 UTC, Paul Backus 
 wrote:
 On Wednesday, 25 August 2021 at 13:23:22 UTC, wjoe wrote:
 [...]
Errors thrown by assertions in unit tests are a special case:
 [...]
Source: https://dlang.org/spec/unittest.html
Also, does that hold for something like assertThrown!SomeError ?
Officially, according to what's written in the spec? No. In practice? Yes. And enough people rely on custom assertions like assertThrown and the ones in unit-threaded [1] that this is unlikely to change. [1] https://unit-threaded.dpldocs.info/unit_threaded.assertions.html
Thanks. I wondered about that for some time.
Aug 25
prev sibling parent user1234 <user1234 12.de> writes:
On Wednesday, 25 August 2021 at 11:46:18 UTC, jfondren wrote:

 [...]
         alias tests = AliasSeq!(__traits(getUnitTests, [...]
drop that useless AliasSeq please ;)
Aug 25
prev sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven 
Schveighoffer wrote:
 Disagree. I'm fine with the simple unittest experience, and 
 avoiding putting all this kind of bloat into the runtime.

 -Steve
What bloat ? Is it the amount of code needed, or the binary size that bothers you ? For the former, we already have many framework doing this, so there's a real interest in having it, and having it in a single place means that improvements will be centralized, instead of being duplicated. So, less code bloat when you look at the big picture. For the later, I've never seen anyone bother about unittest binary size. I don't think we should. And if the unittest framework code ends up in the binary when `-unittest` is not used, we're doing it wrong.
Aug 25
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/26/21 12:12 AM, Mathias LANG wrote:
 On Wednesday, 25 August 2021 at 11:11:24 UTC, Steven Schveighoffer wrote:
 Disagree. I'm fine with the simple unittest experience, and avoiding 
 putting all this kind of bloat into the runtime.
What bloat ? Is it the amount of code needed, or the binary size that bothers you ?
unittesting code is in the binary whether unittests are run or not. Whether unittests should be run is all decided at runtime. But also, what I meant is, if you are going to include all the parts to colorize things, that now needs to be part of druntime.
 For the former, we already have many framework doing this, so there's a 
 real interest in having it, and having it in a single place means that 
 improvements will be centralized, instead of being duplicated. So, less 
 code bloat when you look at the big picture.
This is different, because the language doesn't have to include the framework to do it. You can write this separately.
 For the later, I've never seen anyone bother about unittest binary size. 
 I don't think we should. And if the unittest framework code ends up in 
 the binary when `-unittest` is not used, we're doing it wrong.
unittest framework code exists whether we are unittesting or not. And yet more seemingly unrelated "druntime" modules need to be built. -Steve
Aug 26
parent reply Mathias LANG <geod24 gmail.com> writes:
On Friday, 27 August 2021 at 00:44:03 UTC, Steven Schveighoffer 
wrote:
 unittesting code is in the binary whether unittests are run or 
 not. Whether unittests should be run is all decided at runtime.
And that is the root of the problem. Very few people expect or want this. Let's just make `-unittest` its own, separate compilation mode, with a different entry point, like it should have been done 15 years ago.
 But also, what I meant is, if you are going to include all the 
 parts to colorize things, that now needs to be part of druntime.
Considering the above is fixed, this isn't a problem anymore. Either via `--gc-sections` or similar, or simply by making it a compiler-instantiated template.
 This is different, because the language doesn't have to include 
 the framework to do it. You can write this separately.
I like that D allows me to do pretty much anything the way I want it. I can override operators to the extent that I can event replicate the special case that is associative array first assignment. That's pure, plain amazing. And it comes without bloat. No runtime indirection, things are "pay-as-you-go". But when I throw in the `-unittest` switch (or the `-cov` or `-profile` for that matter), I want all the bells and whistles. Give me colors, an ncurses interface, an HTTP server even! Give me the full battery pack. Don't expect everyone and their mother to implement their own testing framework, that's just bonkers.
Aug 27
parent reply Johan <j j.nl> writes:
On Friday, 27 August 2021 at 07:48:40 UTC, Mathias LANG wrote:
 But when I throw in the `-unittest` switch (or the `-cov` or 
 `-profile` for that matter), I want all the bells and whistles. 
 Give me colors, an ncurses interface, an HTTP server even! Give 
 me the full battery pack. Don't expect everyone and their 
 mother to implement their own testing framework, that's just 
 bonkers.
Why would you expect a _compiler_ to implement a testing framework? If the compiler implements the testing framework, any improvement of the testing framework would require an update of the compiler, which is a _very_ large hurdle. (You want colors? You get 50 new language semantic changes for free!) I think it's much better if the testing framework is implementated separate from anything else. That doesn't mean everybody has to implement it from scratch, just that they should download it from some other location than the compiler website. Separation of concerns. (by the way, I'm convinced `unittest` is a misfeature of the language. Besides the clear downside that is apparent from this thread, it hurts legibility and because of that it actually discourages extensive unittesting) -Johan
Aug 27
next sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
 On Friday, 27 August 2021 at 07:48:40 UTC, Mathias LANG wrote:
 But when I throw in the `-unittest` switch (or the `-cov` or 
 `-profile` for that matter), I want all the bells and 
 whistles. Give me colors, an ncurses interface, an HTTP server 
 even! Give me the full battery pack. Don't expect everyone and 
 their mother to implement their own testing framework, that's 
 just bonkers.
Why would you expect a _compiler_ to implement a testing framework?
I don't. I expect the _runtime_ to do it. And to give me the tool to override it if I have needs that aren't covered by it. Currently it only does the later.
Aug 27
parent reply Johan <j j.nl> writes:
On Friday, 27 August 2021 at 10:59:54 UTC, Mathias LANG wrote:
 On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
 On Friday, 27 August 2021 at 07:48:40 UTC, Mathias LANG wrote:
 But when I throw in the `-unittest` switch (or the `-cov` or 
 `-profile` for that matter), I want all the bells and 
 whistles. Give me colors, an ncurses interface, an HTTP 
 server even! Give me the full battery pack. Don't expect 
 everyone and their mother to implement their own testing 
 framework, that's just bonkers.
Why would you expect a _compiler_ to implement a testing framework?
I don't. I expect the _runtime_ to do it. And to give me the tool to override it if I have needs that aren't covered by it. Currently it only does the later.
For this discussion I consider the compiler+runtime to be a single entity, but OK: same question, why should the language runtime do it? I find it a large stretch to include full unittesting framework inside the scope of a language runtime's core functionality. What is so special about unittesting that requires the implementation to be inside the runtime? The downsides are clear to me (and large), what advantage is gained by having it in the language runtime? -Johan
Aug 27
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/27/21 7:19 AM, Johan wrote:
 On Friday, 27 August 2021 at 10:59:54 UTC, Mathias LANG wrote:
 On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
 On Friday, 27 August 2021 at 07:48:40 UTC, Mathias LANG wrote:
 But when I throw in the `-unittest` switch (or the `-cov` or 
 `-profile` for that matter), I want all the bells and whistles. Give 
 me colors, an ncurses interface, an HTTP server even! Give me the 
 full battery pack. Don't expect everyone and their mother to 
 implement their own testing framework, that's just bonkers.
Why would you expect a _compiler_ to implement a testing framework?
I don't. I expect the _runtime_ to do it. And to give me the tool to override it if I have needs that aren't covered by it. Currently it only does the later.
For this discussion I consider the compiler+runtime to be a single entity, but OK: same question, why should the language runtime do it? I find it a large stretch to include full unittesting framework inside the scope of a language runtime's core functionality. What is so special about unittesting that requires the implementation to be inside the runtime? The downsides are clear to me (and large), what advantage is gained by having it in the language runtime?
separate compilation. The compiler doesn't know all the modules that have been built with unittests. Only the runtime does. The compiler embeds unittests into the ModuleInfo, which means after the linking is done, then you can run unittests if there are any. But the unittest "framework" right now is really simple (it's in fact [this function](https://github.com/dlang/druntime/blob/e6caaab9d359198b760c698dcb6d253afb3f81f6/src/core/r ntime.d#L544-L655). I think it should stay that way, and if you want it more complex, use your own test system. I spent a good deal of time making it easier to install your own handler (in addition to NOT running main after unittests by default). One thing that could be useful is a user-specified way to generate unittest metadata. Right now, you just get a single function pointer in ModuleInfo. You lose any UDAs or references to individual unittest functions. Allowing one to specify a metadata collector on unittests would be very useful in separating the unittest framework from the dependencies, and making it much easier to select a different one, regardless of your dependency preferences. -Steve
Aug 27
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 27 August 2021 at 14:13:56 UTC, Steven Schveighoffer 
wrote:
 separate compilation. The compiler doesn't know all the modules 
 that have been built with unittests. Only the runtime does.
The is another extremely weird design. When you run the unitests, you want to run the unitests for the module you specifically asked for, not for half the galaxy.
Aug 31
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Aug 31, 2021 at 11:08:46AM +0000, deadalnix via Digitalmars-d wrote:
 On Friday, 27 August 2021 at 14:13:56 UTC, Steven Schveighoffer wrote:
 separate compilation. The compiler doesn't know all the modules that
 have been built with unittests. Only the runtime does.
 
The is another extremely weird design. When you run the unitests, you want to run the unitests for the module you specifically asked for, not for half the galaxy.
Yeah, this problem has been brought up before, but nobody had the solution. The problem is that when you compile with -unittest, it turns on unittests for ALL modules, including Phobos, 3rd party libraries, *everything*. This is rarely what you want -- Phobos, for example, contains a bunch of extremely heavy duty unittests that end users don't ever want to run. Because of this, the version=StdUnittest hack was implemented in Phobos. But this is not scalable: every library and his neighbour's dog would have to implement their own unittest hack version identifier for this scheme to work. And if some 3rd party library author neglected to do this, you're left out in the cold. I think it makes much more sense to only compile unittests for modules explicitly specified on the command-line. After all, the user is unlikely to be interested in unittesting 3rd party libraries; his primary interest is to unittest his *own* code. However, this simplistic approach falls flat in the face of `dmd -i`. Schemes like `-unittest=mod1,mod2,...` have been proposed before, but AFAIK never implemented. T -- If it's green, it's biology, If it stinks, it's chemistry, If it has numbers it's math, If it doesn't work, it's technology.
Aug 31
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/31/21 12:22 PM, H. S. Teoh wrote:
 On Tue, Aug 31, 2021 at 11:08:46AM +0000, deadalnix via Digitalmars-d wrote:
 On Friday, 27 August 2021 at 14:13:56 UTC, Steven Schveighoffer wrote:
 separate compilation. The compiler doesn't know all the modules that
 have been built with unittests. Only the runtime does.
The is another extremely weird design. When you run the unitests, you want to run the unitests for the module you specifically asked for, not for half the galaxy.
This is exactly how it works. You ask to build unittests for a module by specifying `-unittest` on the build line for that module. The linker hooks up the module infos into a data segment, and then the runtime looks through those looking for ones with unittests to run. So for instance, if you link against library X, and you didn't actually build library X with -unittest, then you won't run library X's unittests, only the ones in your project. But my point is, the reason it's done this way is because D uses the build system that C uses -- which has a compiler and a linker as separate steps. It's also the reason that cycle detection can't be run until program startup.
 
 Yeah, this problem has been brought up before, but nobody had the
 solution.
 
 The problem is that when you compile with -unittest, it turns on
 unittests for ALL modules, including Phobos, 3rd party libraries,
 *everything*.  This is rarely what you want -- Phobos, for example,
 contains a bunch of extremely heavy duty unittests that end users don't
 ever want to run.  Because of this, the version=StdUnittest hack was
 implemented in Phobos.  But this is not scalable: every library and his
 neighbour's dog would have to implement their own unittest hack version
 identifier for this scheme to work.  And if some 3rd party library
 author neglected to do this, you're left out in the cold.
There is some misunderstandings with what you say here. When you compile with `-unittest`, all unittests are turned on BUT only unittests in modules given on the command line are semantically analyzed and put into object files. So you don't get all the unittests from 3rd party libraries in your build. However, it does turn on the `version(unittest)`, which might make some libraries alter how their types are defined. It also will include unittests inside templates that you instantiate (because it doesn't know if those were already done somewhere). This is why I very much dislike unittests inside templates. The `version(StdUnittest)` was added so that extra imports, altered and extra types were not included, as well as removing unittests inside templates. It wasn't added to exclude standard unittests (which was already happening). -Steve
Aug 31
parent reply deadalnix <deadalnix gmail.com> writes:
On Tuesday, 31 August 2021 at 16:50:07 UTC, Steven Schveighoffer 
wrote:
 So for instance, if you link against library X, and you didn't 
 actually build library X with -unittest, then you won't run 
 library X's unittests, only the ones in your project.
So I was curious at to what was meant by this. Let me try to make it explicit why the current behavior is nonsense. I ran something akin to: ``` $ dmd -m64 -Isrc -w -debug -g -main -unittest -i -run src/somemodule.d 3 modules passed unittests ``` It is clear that this is running random unitests I don't care about. Module that are brought in via the -i flag are also compiled with their unittests, which is nonsense. On the other hand, I made an interesting discovery: ``` $ dmd -m64 -Isrc -w -debug -g -main -unittest -i -run src/module1.d src/module2.d ``` Will run exclusively the tests for module1.d and ignore module2.d , which is considered an argument to be passed to the main that don't exist and is never run. This is complete nonsense (and means that I simply run a fraction of my unitests and have no idea how to fix my build to runt hem all). You'll note that I didn't find a way to get rdmd to do the right thing here, so I'm recompiling everything all the time for no good reason when i want to run the tests, which is also a giant pain in the ass.
 But my point is, the reason it's done this way is because D 
 uses the build system that C uses -- which has a compiler and a 
 linker as separate steps.
I don't think this is a good reason. There is nothing in the C style build.link model that prevent us from doing the right thing.
 It's also the reason that cycle detection can't be run until 
 program startup.
Cool, this can be done at startup from module infos. The problem is not at this level. The compiler provide set of flags that provide a sensible behavior. it's all nonsense.
Sep 02
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Thursday, 2 September 2021 at 17:26:46 UTC, deadalnix wrote:
 ```
 $ dmd -m64 -Isrc -w -debug -g -main -unittest -i -run 
 src/somemodule.d
 3 modules passed unittests
 ```

 It is clear that this is running random unitests I don't care 
 about. Module that are brought in via the -i flag are also 
 compiled with their unittests, which is nonsense.
This is the only reasonable outcome from this combination of flags. Not everything that ever surprises you is nonsense. If you're not familiar with how dmd works, then being surprised is just part of learning how it works.
 On the other hand, I made an interesting discovery:

 ```
 $ dmd -m64 -Isrc -w -debug -g -main -unittest -i -run 
 src/module1.d src/module2.d
 ```

 Will run exclusively the tests for module1.d and ignore 
 module2.d , which is considered an argument to be passed to the 
 main that don't exist and is never run.
Yep. That's what -run does. It passes the remaining commandline to the program to be run.
 This is complete nonsense
The only *conceivable* alternative to this "complete nonsense" is to always require a -- separator, which dub does, which is more an artifact of getopt-style commandline processing than the outcome of a thoughtfully designed CLI. dub requiring -- just means I never use it directly. Here's my two-line ~/.local/bin/dfmt: ``` dub run -q --skip-registry=all dfmt -- -i --brace_style=otbs "$ " ```
 (and means that I simply run a fraction of my unitests and have 
 no idea how to fix my build to runt hem all).
``` $ grep . *.d alsotest.d:unittest { alsotest.d: assert(!0); alsotest.d:} donttest.d:unittest { donttest.d: assert(0); donttest.d:} pleasetest.d:unittest { pleasetest.d: assert(true); pleasetest.d:} $ dmd -c donttest.d; dmd -c -unittest pleasetest.d alsotest.d $ dmd -main -of=main *.o $ ./main 2 modules passed unittests ```
Sep 02
parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 2 September 2021 at 17:51:10 UTC, jfondren wrote:
 This is the only reasonable outcome from this combination of 
 flags. Not everything that ever surprises you is nonsense. If 
 you're not familiar with how dmd works, then being surprised is 
 just part of learning how it works.
No, it clearly isn't. In fact, there are exactly 0 use case for what this combination of flag is doing and 0 combination of flags for use cases that exist in the wild. If you don't see the problem, I don't know what to tell you. Maybe "Please do not design any software that is meant to be used by anyone ever, thanks".
 Yep. That's what -run does. It passes the remaining commandline 
 to the program to be run.

 This is complete nonsense
The only *conceivable* alternative to this "complete nonsense" is to always require a -- separator, which dub does, which is more an artifact of getopt-style commandline processing than the outcome of a thoughtfully designed CLI.
Once again, snap out of whatever mindset you are in. This is *OBVIOUSLY* nonsense because there is *NO PROGRAM TO BE RUN*. The compiler know this because I had to pass the -main flag to begin with. There is *NO MAIN* there is *NO EXPECTATION THAT ANY MAIN IS RUN* so passing argument to it simply does not make sense. As Pauli would say, it's not even wrong. Think of it this way. There is a unitest feature in the language. There are literally no way to tell the compiler "please run the unitests for this set of module". Or if there is one, it is so bizantine that nobody seems to be able to figure it out. This is a 7 pages long thread about it, full of D experts, and yet, nobody knows how to do this simple task.
Sep 02
parent reply jfondren <julian.fondren gmail.com> writes:
On Thursday, 2 September 2021 at 18:01:34 UTC, deadalnix wrote:
 On Thursday, 2 September 2021 at 17:51:10 UTC, jfondren wrote:
 This is the only reasonable outcome from this combination of 
 flags. Not everything that ever surprises you is nonsense. If 
 you're not familiar with how dmd works, then being surprised 
 is just part of learning how it works.
No, it clearly isn't. In fact, there are exactly 0 use case for what this combination of flag is doing and 0 combination of flags for use cases that exist in the wild.
1 use case: run all the tests in all the imported modules. The thing it does.
 This is *OBVIOUSLY* nonsense because there is *NO PROGRAM TO BE 
 RUN*. The compiler know this because I had to pass the -main 
 flag to begin with.
You want -run to treat the commandline completely differently depending on whether -main has been passed? That would help you to have not been surprised in this one case, but by making the CLI more complex it's just going to be more confusing to more people in the long run. Bug report: I removed -main and suddenly was only testing a single module instead of the several I'd been testing before.
 Think of it this way. There is a unitest feature in the 
 language. There are literally no way to tell the compiler 
 "please run the unitests for this set of module".
This is fair. There's a lot of machinery set up *around* testing but there isn't a one-shot command. 'dub test' also isn't such a command, as it's package-oriented rather than module-oriented.
 Or if there is one, it is so bizantine that nobody seems to be 
 able to figure it out. This is a 7 pages long thread about it, 
 full of D experts, and yet, nobody knows how to do this simple 
 task.
I just demonstrated compiling three modules to only test two of them. This 7 pages long thread has all kinds of talk in it; it is not full of people asking "I have these files; how do I run only these tests in them?" or it'd be full of as many answers. The machinery built up *around* testing is very good and can be persuaded to work any way you want.
Sep 02
parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 2 September 2021 at 18:15:17 UTC, jfondren wrote:
 You want -run to treat the commandline completely differently 
 depending on whether -main has been passed? That would help you 
 to have not been surprised in this one case, but by making the 
 CLI more complex it's just going to be more confusing to more 
 people in the long run.
No. I want a way to run the unitests in a set of module, and only these modules. I don't care about `-run`, `-main` or whatever. I care about the fact that there is no sensible combination of these flags that can achieve this.
 I just demonstrated compiling three modules to only test two of 
 them. This 7 pages long thread has all kinds of talk in it; it 
 is not full of people asking "I have these files; how do I run 
 only these tests in them?" or it'd be full of as many answers. 
 The machinery built up *around* testing is very good and can be 
 persuaded to work any way you want.
This is certainly possible. Nobody ever argued this wasn't possible. What was argued is that you pretty much have to rollout your own test framework to do this, and that it doesn't compose well when you import 3rd party code, which your demonstration confirms. The machinery built around testing is absolutely not very good. You can claim it is, but the very fact that you have to raise scafolding that do not compose to do the most basic thing is a demonstration that it isn't. You simply can't argue with facts. The fact is that all the functionality that are desired to do something sensible exist. There is just no way to plug them into each other in any sensible way because all the utilities and flags provided do some weird shit that is kinda useful but not really.
Sep 02
parent reply jfondren <julian.fondren gmail.com> writes:
On Thursday, 2 September 2021 at 18:35:48 UTC, deadalnix wrote:
 The fact is that all the functionality that are desired to do 
 something sensible exist. There is just no way to plug them 
 into each other in any sensible way because all the utilities 
 and flags provided do some weird shit that is kinda useful but 
 not really.
The fact is that dmd is a compiler and not a build system. It therefore has flags that have simple, dumb, easy to compose behaviors rather than too-smart ones that would ever entertain considerations like "There is NO MAIN there is NO EXPECTATION THAT ANY MAIN IS RUN so ---" or "who would ever want what this combination of flags clearly does?" When you pass -main, dmd adds a module to your build that includes a main function, job done. When you pass -unittest, dmd compiles in unittests and (as of recent versions) even has a somewhat nice default testing framework that provides feedback and skips running main(). When you pass --version, dmd prints out the version number without even asking if the commandline also asked for other stuff to be done. The alternative to simple dumb flags is not a paradise where dmd always does what everyone expects; it's its own confusing hell where people are constantly surprised by behaviors that are too complex for the outcome to ever be predicted without a week-long course, instead of a quick check of the current manpage where each flag only needs a single sentence for a description. I think there's an argument to be made that maybe a 'dmd test' should be added to run the tests of a single module. This wouldn't make anything easy that wasn't easy already, but it's a defensible bit of streamlining and it'd support one of D's advantages as a language. I think the very first complaint that someone would raise about it, in all caps, is that the use case of "dmd test a; dmd test b; dmd test c" would incur a lot of recompilation when these modules depend on each other, e.g. c might be built twice without unit tests and then a third time with them.
Sep 02
next sibling parent jfondren <julian.fondren gmail.com> writes:
On Thursday, 2 September 2021 at 19:16:38 UTC, jfondren wrote:
 The fact is that dmd is a compiler and not a build system.
...
 I think there's an argument to be made that maybe a 'dmd test' 
 should be added to run the tests of a single module.
Nah, this discussion is meandering due to some frustration and some specific complaints taking the place of the original complaint. If anyone's following this thread, this is a good time to go back to the original post: https://forum.dlang.org/post/rgalgunhtubgeoyrfwxd forum.dlang.org Where the use case is roughly 1. the developer is knee-deep in a big D project with a bunch of dependencies and modules 2. the developer is doing work on a single module 3. the developer wants to run that module's tests to confirm that 4. the developer wants to do this without incurring a "testing!" variant of https://xkcd.com/303/ and without needing a bunch of boilerplate like Unittest_ThisModule version blocks in each module.
Sep 02
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 2 September 2021 at 19:16:38 UTC, jfondren wrote:
 The alternative to simple dumb flags is not a paradise where 
 dmd always does what everyone expects; it's its own confusing 
 hell where people are constantly surprised by behaviors that 
 are too complex for the outcome to ever be predicted without a 
 week-long course, instead of a quick check of the current 
 manpage where each flag only needs a single sentence for a 
 description.
This is not the alternative. This what exists today.
Sep 02
parent deadalnix <deadalnix gmail.com> writes:
On Thursday, 2 September 2021 at 20:46:57 UTC, deadalnix wrote:
 On Thursday, 2 September 2021 at 19:16:38 UTC, jfondren wrote:
 The alternative to simple dumb flags is not a paradise where 
 dmd always does what everyone expects; it's its own confusing 
 hell where people are constantly surprised by behaviors that 
 are too complex for the outcome to ever be predicted without a 
 week-long course, instead of a quick check of the current 
 manpage where each flag only needs a single sentence for a 
 description.
This is not the alternative. This what exists today.
Case in point: https://issues.dlang.org/show_bug.cgi?id=22268
Sep 02
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/2/21 1:26 PM, deadalnix wrote:
 On Tuesday, 31 August 2021 at 16:50:07 UTC, Steven Schveighoffer wrote:
 So for instance, if you link against library X, and you didn't 
 actually build library X with -unittest, then you won't run library 
 X's unittests, only the ones in your project.
So I was curious at to what was meant by this. Let me try to make it explicit why the current behavior is nonsense. I ran something akin to: ``` $ dmd -m64 -Isrc -w -debug -g -main -unittest -i -run src/somemodule.d 3 modules passed unittests ``` It is clear that this is running random unitests I don't care about. Module that are brought in via the -i flag are also compiled with their unittests, which is nonsense.
The -i flag finds all dependencies and builds them all at once as if you weren't linking in the object code that is in a library. That's not what I was saying. What I was saying is, you have a library, you built it without -unittest (on its own build line, or just use dub because it does the right thing), and then you build your main application with -unittest, linking in the already-built library (or its object files). The runtime runs all unittests it finds, and so it's running just the modules that were built with the -unittest flag. When you use -i, it means *ALL MODULES AND IMPORTS* are built with the -unittest flag, and so they all run.
 
 Will run exclusively the tests for module1.d and ignore module2.d , 
 which is considered an argument to be passed to the main that don't 
 exist and is never run.
 
 This is complete nonsense (and means that I simply run a fraction of my 
 unitests and have no idea how to fix my build to runt hem all).
-i is going to run all unittests. How else would it interpret the command line? If you want to build some modules differently than others (i.e. some with unittests and some without), you can't use -i.
 But my point is, the reason it's done this way is because D uses the 
 build system that C uses -- which has a compiler and a linker as 
 separate steps.
I don't think this is a good reason. There is nothing in the C style build.link model that prevent us from doing the right thing.
What is the "right thing"? Reading what you expect in your mind is not what a compiler can do. Define how it will work. I think unittests work acceptably the way they are now, but I'm sure others would be keen to have some more bells and whistles. Just make a proposal and get it merged in. Start by saying "when I type `dmd -m64 -Isrc -w -debug -g -main -unittest -i -run src/somemodule.d`, here is what I expect to happen: ...", and then define rules that will make that happen. -Steve
Sep 02
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 2 September 2021 at 23:00:06 UTC, Steven 
Schveighoffer wrote:
 When you use -i, it means *ALL MODULES AND IMPORTS* are built 
 with the -unittest flag, and so they all run.
I know. That's dumb.
 -i is going to run all unittests. How else would it interpret 
 the command line? If you want to build some modules differently 
 than others
 (i.e. some with unittests and some without), you can't use -i.
Is this groundhog day? There are no combination of flags that do the sensible thing. I can't use -i, no problem. I can't use anything else either, big problem. The proof is in the pudding, you just told me that I need a full build infra to run unittests for something that use anything else than phobos.
 What is the "right thing"? Reading what you expect in your mind 
 is not what a compiler can do. Define how it will work. I think 
 unittests work acceptably the way they are now, but I'm sure 
 others would be keen to have some more bells and whistles. Just 
 make a proposal and get it merged in.
The right thing is being able to run the unitests of a god damn module easily. not the unit tests of that module and all of its dependencies. Not using a complex test framework. Just running the unit tests of a module. This is the sensible thing. This is the simplest thing. This is the "right thing". It doesn't matter if the module has a main or not. It doesn't matter if the module depends on phobos or half the solar system.
 Start by saying "when I type `dmd -m64 -Isrc -w -debug -g -main 
 -unittest -i -run src/somemodule.d`, here is what I expect to 
 happen: ...", and then define rules that will make that happen.
Here is what I expect. $ dmd --run-the-god-damn-unittest mymodule.d And it run the unit tests of mymodule.d . Replace `--run-the-god-damn-unittest` by any set of flags that pleases you, for as long as there is one.
Sep 02
next sibling parent reply Adam Ruppe <destructionator gmail.com> writes:
The -unittest flag should just accept a pattern, same as -i, so 
you can explicitly add or remove things that get the unittest 
build.

Then you can be like dmd -unittest=foo.bar -i main.d and it will 
build foo.bar with unitest and skip the rest while still 
auto-following imports.
Sep 02
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Sep 02, 2021 at 11:48:25PM +0000, Adam Ruppe via Digitalmars-d wrote:
 The -unittest flag should just accept a pattern, same as -i, so you
 can explicitly add or remove things that get the unittest build.
 
 Then you can be like dmd -unittest=foo.bar -i main.d and it will build
 foo.bar with unitest and skip the rest while still auto-following
 imports.
Haha, I just wrote a post describing pretty much the same thing. Great minds think alike. ;-) T -- You are only young once, but you can stay immature indefinitely. -- azephrahel
Sep 02
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Sep 02, 2021 at 11:22:57PM +0000, deadalnix via Digitalmars-d wrote:
[...]
 Here is what I expect.
 
 $ dmd --run-the-god-damn-unittest mymodule.d
 
 And it run the unit tests of mymodule.d . Replace
 `--run-the-god-damn-unittest` by any set of flags that pleases you,
 for as long as there is one.
I propose to extend the -unittest flag to take an optional list of identifiers: -unittest=ident1,ident2,... Each identifier is either a module or package, in which case it means compile unittests for the listed module(s) or all modules whose FQN has a prefix matching the identifier; or one of these special identifiers: . Compile unittests only for modules whose source files are found in the current working directory or one of its descendents; default Compile *all* unittests in all modules being compiled (the current default behaviour). module Compile unittests for all modules explicitly specified on the command-line. (I deliberately picked reserved keywords and symbols which cannot be confused with D module names.) So what Amaury wants would be accomplished by: dmd -unittest=module -main -i -run mymodule.d To compile and run local unittests (i.e., everything contained in the current directory or one of its recursive subdirectories, excluding things outside like Phobos or 3rd party dub dependencies): dmd -unittest=. -main -i -run myprogram.d To compile and run unittests for a specific packages/modules (including a 3rd party library if for whatever strange reason you need to): dmd -unittest=path.to.package -i -run myprogram.d To compile and run unittests for multiple packages/modules: dmd -unittest=std,thirdparty.pkg,mysubproject -i -run myprogram.d Finally, if no argument is given to -unittest, then `-unittest=default` is assumed, which behaves like things do today. Notes: The above specified filters on modules only apply to the set of modules currently being compiled; that's why -i is necessary if your module imports anything else. Would this satisfy everybody? T -- "Hi." "'Lo."
Sep 02
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 3 September 2021 at 00:09:37 UTC, H. S. Teoh wrote:
 Would this satisfy everybody?
1/ make module the default. This is really the sensible behavior. Changing imports really change the list of test that are run, especially not recursively. 2/ the `-main` problem remains. 3/ it still isn't possible to run the unit tests of several modules in one go by passing a list of modules to the compiler (in the same way you can compile several modules in one go and generate an object file for all of them at once).
Sep 02
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Sep 03, 2021 at 01:12:13AM +0000, deadalnix via Digitalmars-d wrote:
 On Friday, 3 September 2021 at 00:09:37 UTC, H. S. Teoh wrote:
 Would this satisfy everybody?
 
1/ make module the default. This is really the sensible behavior. Changing imports really change the list of test that are run, especially not recursively.
Making module the default would break existing usage. But whatever, the important thing is to implement -unittest=xxx first, then changing the default is a 1-line change later.
 2/ the `-main` problem remains.
Even though it's kinda related, -main is really an orthogonal issue. I've also run into the problem myself where -main is needed when your modules don't contain main() but must be suppressed or worked around when it does declare main(). This should be fixed so that -main *optionally* inserts an empty main(), e.g., as a weak symbol, or whatever, so that the existence of a real main() doesn't cause a linker error.
 3/ it still isn't possible to run the unit tests of several modules in
 one go by passing a list of modules to the compiler (in the same way
 you can compile several modules in one go and generate an object file
 for all of them at once).
Of course you can. Under my proposal: dmd -unittest=module -i -run mod1.d mod2.d mod3.d main.d T -- Democracy: The triumph of popularity over principle. -- C.Bond
Sep 02
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 3 September 2021 at 03:06:27 UTC, H. S. Teoh wrote:
 2/ the `-main` problem remains.
Even though it's kinda related, -main is really an orthogonal issue. I've also run into the problem myself where -main is needed when your modules don't contain main() but must be suppressed or worked around when it does declare main(). This should be fixed so that -main *optionally* inserts an empty main(), e.g., as a weak symbol, or whatever, so that the existence of a real main() doesn't cause a linker error.
Well, yes and no. It can be solved in an orthogonal fashion, but this really is the same problem: the flag you can pass to the compiler are doing nonsensical stuff and are not amendable to common use cases.
 3/ it still isn't possible to run the unit tests of several 
 modules in one go by passing a list of modules to the compiler 
 (in the same way you can compile several modules in one go and 
 generate an object file for all of them at once).
Of course you can. Under my proposal: dmd -unittest=module -i -run mod1.d mod2.d mod3.d main.d
No, this will run the code in mod1.d , passing ["mod2.d" "mod3.d" "main.d"] as command line arguments.
Sep 03
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Sep 03, 2021 at 08:39:35AM +0000, deadalnix via Digitalmars-d wrote:
[...]
 3/ it still isn't possible to run the unit tests of several
 modules in one go by passing a list of modules to the compiler (in
 the same way you can compile several modules in one go and
 generate an object file for all of them at once).
Of course you can. Under my proposal: dmd -unittest=module -i -run mod1.d mod2.d mod3.d main.d
No, this will run the code in mod1.d , passing ["mod2.d" "mod3.d" "main.d"] as command line arguments.
Oh right, this is the other bogonity of -run: it expects the following argument to be the module that gets run. I really don't understand the logic of this: surely the compiler ought to be able to know which module main() is declared in, so why does it need the user to select one specific module to be run? This should be automated within the compiler itself. T -- Never criticize a man until you've walked a mile in his shoes. Then when you do criticize him, you'll be a mile away and he won't have his shoes.
Sep 05
parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 5 September 2021 at 15:04:58 UTC, H. S. Teoh wrote:
 On Fri, Sep 03, 2021 at 08:39:35AM +0000, deadalnix via 
 Digitalmars-d wrote: [...]
 3/ it still isn't possible to run the unit tests of 
 several modules in one go by passing a list of modules to 
 the compiler (in the same way you can compile several 
 modules in one go and generate an object file for all of 
 them at once).
Of course you can. Under my proposal: dmd -unittest=module -i -run mod1.d mod2.d mod3.d main.d
No, this will run the code in mod1.d , passing ["mod2.d" "mod3.d" "main.d"] as command line arguments.
I really don't understand the logic of this: surely the compiler ought to be able to know which module main() is declared in, so why does it need the user to select one specific module to be run?
``` $ grep -H . mod?.d mod1.d:void main(string[] args) { mod1.d: import std.stdio : writeln; mod1.d: writeln(args[1 .. $]); mod1.d:} $ dmd -run mod1.d mod2.d mod3.d ["mod2.d", "mod3.d"] ``` mod1.d is a program that prints its args that I'd like to run with some arguments. mod2.d and mod3.d are not modules that even exist; I just want them printed out like that. Distinguishing between dmd and program arguments isn't a matter of compiler smarts.
Sep 05
next sibling parent jfondren <julian.fondren gmail.com> writes:
On Sunday, 5 September 2021 at 15:52:51 UTC, jfondren wrote:
 On Sunday, 5 September 2021 at 15:04:58 UTC, H. S. Teoh wrote:
 I really don't understand the logic of this: surely the 
 compiler ought to be able to know which module main() is 
 declared in, so why does it need the user to select one 
 specific module to be run?
``` $ grep -H . mod?.d mod1.d:void main(string[] args) { mod1.d: import std.stdio : writeln; mod1.d: writeln(args[1 .. $]); mod1.d:} $ dmd -run mod1.d mod2.d mod3.d ["mod2.d", "mod3.d"] ``` mod1.d is a program that prints its args that I'd like to run with some arguments. mod2.d and mod3.d are not modules that even exist; I just want them printed out like that. Distinguishing between dmd and program arguments isn't a matter of compiler smarts.
More: ``` $ grep -H . mod?.d mod1.d:void main(string[] args) { mod1.d: import std.stdio : writeln; mod1.d: writeln(args[1 .. $]); mod1.d:} mod2.d:shared static this() { mod2.d: import std.stdio : writeln; mod2.d: writeln("module init: mod2.d"); mod2.d:} mod3.d:shared static this() { mod3.d: import std.stdio : writeln; mod3.d: writeln("module init: mod3.d"); mod3.d:} $ dmd -run mod1.d mod2.d mod3.d ["mod2.d", "mod3.d"] $ dmd mod2.d mod3.d -run mod1.d mod2.d mod3.d module init: mod2.d module init: mod3.d ["mod2.d", "mod3.d"] $ dmd mod2.d mod1.d -run mod3.d mod2.d mod1.d module init: mod2.d module init: mod3.d ["mod2.d", "mod1.d"] $ echo 'import mod2, mod3;' >> mod1.d $ dmd -i -run mod1.d module init: mod2.d module init: mod3.d [] ``` In particular: ``` $ dmd mod2.d mod3.d -run mod1.d mod2.d mod3.d $ dmd mod2.d mod1.d -run mod3.d mod2.d mod1.d ``` dmd links all the objects together to make an executable and runs it, and the behavior's the same with main()-having mod1.d and main()-lacking mod3.d as the argument after -run.
Sep 05
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Sep 05, 2021 at 03:52:51PM +0000, jfondren via Digitalmars-d wrote:
 On Sunday, 5 September 2021 at 15:04:58 UTC, H. S. Teoh wrote:
 On Fri, Sep 03, 2021 at 08:39:35AM +0000, deadalnix via Digitalmars-d
 wrote: [...]
 	dmd -unittest=module -i -run mod1.d mod2.d mod3.d main.d
No, this will run the code in mod1.d , passing ["mod2.d" "mod3.d" "main.d"] as command line arguments.
I really don't understand the logic of this: surely the compiler ought to be able to know which module main() is declared in, so why does it need the user to select one specific module to be run?
``` $ grep -H . mod?.d mod1.d:void main(string[] args) { mod1.d: import std.stdio : writeln; mod1.d: writeln(args[1 .. $]); mod1.d:} $ dmd -run mod1.d mod2.d mod3.d ["mod2.d", "mod3.d"] ``` mod1.d is a program that prints its args that I'd like to run with some arguments. mod2.d and mod3.d are not modules that even exist; I just want them printed out like that. Distinguishing between dmd and program arguments isn't a matter of compiler smarts.
This is seriously a wrong conflation of the program arguments with a program module. If the whole point is to support specifying the command-line to the program, then the argument list should appear immediately after -run; the first argument after -run should not be conflated with argv[0]. (For example, what if I'm writing an analogue of busybox, and need to specify a different argv[0] from the name of the executable?) The current behaviour is counterintuitive, doesn't cover all use cases, and has inconsistencies that lead to confusion. It's poor design. T -- I am a consultant. My job is to make your job redundant. -- Mr Tom
Sep 05
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Sunday, 5 September 2021 at 16:12:26 UTC, H. S. Teoh wrote:
 This is seriously a wrong conflation of the program arguments 
 with a program module.  If the whole point is to support 
 specifying the command-line to the program, then the argument 
 list should appear immediately after -run; the first argument 
 after -run should not be conflated with argv[0]. (For example, 
 what if I'm writing an analogue of busybox, and need to specify 
 a different argv[0] from the name of the executable?)

 The current behaviour is counterintuitive, doesn't cover all 
 use cases, and has inconsistencies that lead to confusion.  
 It's poor design.
The current behavior is entirely intuitive, it covers all cases, it's consistent, it's fine. The only difference from it and a lazy getopt solution is lacking a -- before the arguments. The only difference between it and your proposed solution is that -run would no longer require an argument. Listen to yourself: ``` ``` It is absolutely not the case that one of these is wretched bad design and the other is pure and great design. These are completely identical designs. From the perspective of someone who's never seen -run used, it's a coinflip whether they'll guess it works one way or another, and it's no longer a coinflip after they read the manpage or have seen it used--it's just memory at that point, and either design is justifiable and easy to remember. If the current design were counterintuitive, then people would occasionally screw it up even after learning it. I use -run all the time, almost more often than rdmd --eval, and I've never once had a problem. "I want to -run this" even follows SVO grammar. I don't think I'd even be bothered--or notice--if a PR went through that made -run nonpositional, but you obviously don't use this feature all the time, what's with the ludicrously over-the-top vitriol about it?
 the first argument after -run should not be conflated with 
 argv[0]. (For example, what if I'm writing an analogue of 
 busybox, and need to specify a different argv[0] from the name 
 of the executable?)
argv[0] is provided by the kernel, not userspace. `man execve` and `man posix_spawn` and such will have an argv parameter, but that's for subsequent arguments. The most busybox can do is tell the kernel what to run with one path or another, in the extreme by creating a temporary symlink.
Sep 05
parent jfondren <julian.fondren gmail.com> writes:
On Sunday, 5 September 2021 at 16:38:01 UTC, jfondren wrote:
 I don't think I'd even be bothered--or notice--if a PR went 
 through that made -run nonpositional, but you obviously don't 
 use this feature all the time, what's with the ludicrously 
 over-the-top vitriol about it?
Regrettably, avoiding criticizing something after 5 seconds of thought isn't an adequate replacement for thinking it through, either. Because -run still needs to know where the program's parameters begin, it can't be completely non-positional.
Sep 05
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Sunday, 5 September 2021 at 16:12:26 UTC, H. S. Teoh wrote:
 The current behaviour is counterintuitive, doesn't cover all 
 use cases, and has inconsistencies that lead to confusion.  
 It's poor design.
I'm glad that, one by one, people are waking up to this fact. It took 80+ messages, but we're getting there. This is especially bonkers in the unittest case, because we are not even planning to run main.
Sep 05
prev sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Thursday, 2 September 2021 at 23:22:57 UTC, deadalnix wrote:
 The right thing is being able to run the unitests of a god damn 
 module easily.
A patch for silly.d that boils down to ``` string targetmodule; // add "test a single module" to existing getopt flags if (targetmodule && moduleName!module_ != targetmodule) continue; ``` permits this usage: ``` $ time dub -q test -- --module=testme ✓ testme __unittest_L1_C1 Summary: 1 passed, 0 failed in 0 ms real 0m0.190s user 0m0.129s sys 0m0.059s ``` Which tests that single module and ignores the others. This still compiles in all the other modules of your project, still compiles in their unittest blocks, still runs their module initialization, still might rebuild a dub dependency, but in the end it runs only the unittest blocks of the chosen module. Anything short of that is going to be a hacky once-off solution that relies on package-specific knowledge that a module's unittests don't really depend on that other stuff happening. At that level you may as well run 'dmd -main -unittest -run module'. I think this is the best you're going to get. You can try this now by cloning the silly repo, patching it, running `dub add-override silly 1.1.1 .` in the repo, and trying out the usage above in your dub package that's already set up to use silly. There's really nothing to the patch but it's here: https://run.dlang.io/is/smXgof
Is this groundhog day?
Well, there's someone in the thread who keeps complaining about random dmd flags, which both draws defenses of those flags and proposals to change them that won't actually give you the workflow you want (like the -unittest= proposal, that still requires you to tell dmd about where all your modules are, which would still be irritating for dub dependencies).
Sep 02
parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 3 September 2021 at 00:34:01 UTC, jfondren wrote:
 Well, there's someone in the thread who keeps complaining about 
 random dmd flags, which both draws defenses of those flags and 
 proposals to change them that won't actually give you the 
 workflow you want (like the -unittest= proposal, that still 
 requires you to tell dmd about where all your modules are, 
 which would still be irritating for dub dependencies).
I'll be blunt, but this is needed because this whole thread is retarded. The current flag's behavior make no sense. They are not useful. They don't so something sensible. They don't serve typical use cases. The whole thing is completely retarded. Yes, improving unittest as some suggest would be an improvement. But it doesn't address the core problem. The `-main` situation remains retarded. It is still not possible to run the unittest of several modules this way, etc... And here is the root problem: something is complete trash and yet, not only I'm told this is normal and expected, but I'm supposed to be happy with it. By yourself, by Walter, by Steven. You guys are obviously smart, but you aren't looking at this straight or even at all. Or you are drowning in so much trash you don't even see it anymore, like a fish in water. In don't know. But this is alarming. In fact, people went through the length of migrating the whole standard library to use the StdUnittest version to work around the problem I'm pointing. Think about it. people would rather change how all the tests of the standard lib work rather than fix this shit. This is insane. Literally. WAKE UP! This is bad. This is really bad. Stop making excuses. It's time to ask, how did this get this way and what can be done so that this never ever happens again. Consider the following. You work in a car company. The eng team just came up with this new design with the engine bolted on the roof of the car. You tell them "Guys, this does not make sense, nobody is going to want a car with the engine on the roof." and they respond to you "Well, this is how it is, you can get better cooling this way and yada yada yada...", what would you think? That these people need a technical solution? No, these people need an intervention. They are building a car for people to use, right? Well, jfondren, you need an intervention. Stop. Take a break and think. You need to look at this with a fresh eye. You have been in this crap for too long and can't see straight anymore. The set of flags the compiler propose is complete nonsense that do not serve users, it has cause major rewrite in the standard library, and motivated the creating of a new test framework per large project. This needs to stop.
Sep 02
next sibling parent jfondren <julian.fondren gmail.com> writes:
On Friday, 3 September 2021 at 01:08:25 UTC, deadalnix wrote:
 you are drowning in so much trash you don't even see it anymore
...
 WAKE UP!
...
 This is bad. This is really bad. Stop making excuses. It's time 
 to ask, how did this get this way and what can be done so that 
 this never ever happens again.

 Consider the following.
Nah.
Sep 02
prev sibling next sibling parent Dennis <dkorpel gmail.com> writes:
On Friday, 3 September 2021 at 01:08:25 UTC, deadalnix wrote:
 The current flag's behavior make no sense. They are not useful. 
 They don't so something sensible. They don't serve typical use 
 cases.
The behavior is good for [custom test runners](https://forum.dlang.org/post/pmniziegqhwbaqwfrbqz forum.dlang.org) or [incremental compilation of unittests](https://github.com/dlang/phobos/blob/d8c702dd54e9a5fd2d914df1335b664f0fd9a 20/posix.mak#L362). Dub projects can use [silly](https://code.dlang.org/packages/silly) to filter tests, and maybe the default dub test runner should be able to do the same. That wouldn't require a change to the `-unittest` flag though.
 And here is the root problem: something is complete trash and 
 yet, not only I'm told this is normal and expected, but I'm 
 supposed to be happy with it.
I'm sorry to hear you're frustrated with the current situation, but I can't just open a dmd pull request breaking the CLI and dismissing everyone else's use cases and get that merged. If you want to rant a bit to vent your frustration, that's fine by me, but if you want this thread to result in an improvement, you'll need to come up with a specific, feasible proposal.
 In fact, people went through the length of migrating the whole 
 standard library to use the StdUnittest version to work around 
 the problem I'm pointing.
No, see [Steven's post](https://forum.dlang.org/post/sglmk0$33m$1 digitalmars.com).
Sep 03
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/2/21 9:08 PM, deadalnix wrote:
 By yourself, by Walter, by Steven. You guys are obviously smart, but you 
 aren't looking at this straight or even at all. Or you are drowning in 
 so much trash you don't even see it anymore, like a fish in water.
Or I haven't thought about or cared about a problem with unittests ever. I just use `dub test` or `dub -b unittest`, and it works as I expect -- all unittests in my project are run, no unittests in dependencies are run. No trash to drown in. I'm not too concerned with your use cases, as the current system runs just fine for me, and if you don't like it, you can use a fancier test runner that provides more control. I don't see a problem with the existence of external unittest systems. The basic system is designed to allow you complete control if you don't like the default, and the default is fine for most people. I'm kinda done with this thread anyway, about to kill it... -Steve
Sep 03
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 3 September 2021 at 01:08:25 UTC, deadalnix wrote:
 WAKE UP!

 This is bad. This is really bad. Stop making excuses. It's time 
 to ask, how did this get this way and what can be done so that 
 this never ever happens again.

 The set of flags the compiler propose is complete nonsense that 
 do not serve users, it has cause major rewrite in the standard 
 library, and motivated the creating of a new test framework per 
 large project. This needs to stop.
First of all, I fully agree. Second, I was digging out my DMD bug 18843 ("-deps -unittest causes cataclysmic memory usage"), just to throw some more fuel on this trash fire of a debate, and good news: `dmd -deps -unittest helloworld.d` no longer causes cataclysmic memory usage: ``` $ cat helloworld.d import std.stdio; void main() { writeln("Hello World"); } $ dmd --version DMD64 D Compiler v2.097.2 Copyright (C) 1999-2021 by The D Language Foundation, All Rights Reserved written by Walter Bright $ dmd -deps -unittest helloworld.d dmd2/linux/bin64/../../src/phobos/std/stdio.d(578): Error: undefined identifier `testFilename` *snip 20 more semi-identical errors* ``` I don't think I need to add anything here...
Sep 03
parent deadalnix <deadalnix gmail.com> writes:
On Friday, 3 September 2021 at 12:49:05 UTC, FeepingCreature 
wrote:
 Second, I was digging out my DMD bug 18843 ("-deps -unittest 
 causes cataclysmic memory usage"), just to throw some more fuel 
 on this trash fire of a debate, and good news: `dmd -deps 
 -unittest helloworld.d` no longer causes cataclysmic memory 
 usage:

 [...]

 I don't think I need to add anything here...
Well, we could add that even though it cause cataclysmic memory usage, the dependency tracking is not done properly, at least for rdmd: https://issues.dlang.org/show_bug.cgi?id=22268
Sep 03
prev sibling parent reply James Blachly <james.blachly gmail.com> writes:
On 9/2/21 7:00 PM, Steven Schveighoffer wrote:
 What I was saying is, you have a library, you built it without -unittest 
 (on its own build line, or just use dub because it does the right 
 thing), and then you build your main application with -unittest, linking 
 in the already-built library (or its object files). The runtime runs all 
 unittests it finds, and so it's running just the modules that were built 
 with the -unittest flag.
Steve: Sorry to reply a few days late and in the middle of a gargantuan thread, but I wanted some clarification from you on dub just doing the right thing -- Let's suppose I maintain a library L (written in D; has unit tests). Suppose further I have client program P, which uses L. As far as I can tell, if P's dubfile includes library L, and the user execeutes `dub test` then the unit tests from the library L as well as program P are built and executed. We had to wrap our library's unittests in `debug(library_unittest)` to escape this behavior. Can you provide some clarity because it sounds like you are saying dub will build P's unit tests, but not library L's? Not at all my experience, unless it was updated recently.
Sep 08
parent jfondren <julian.fondren gmail.com> writes:
On Thursday, 9 September 2021 at 00:13:00 UTC, James Blachly 
wrote:
 Let's suppose I maintain a library L (written in D; has unit 
 tests). Suppose further I have client program P, which uses L.

 As far as I can tell, if P's dubfile includes library L, and 
 the user execeutes `dub test` then the unit tests from the 
 library L as well as program P are built and executed.

 We had to wrap our library's unittests in 
 `debug(library_unittest)` to escape this behavior.

 Can you provide some clarity because it sounds like you are 
 saying dub will build P's unit tests, but not library L's? Not 
 at all my experience, unless it was updated recently.
Either one of the stated behaviors can be the case. Consider these pair of libraries: ``` $ grep -R . ell/dub.sdl:name "ell" ell/dub.sdl:description "A minimal D application." ell/dub.sdl:authors "Julian Fondren" ell/dub.sdl:copyright "Copyright © 2021, Julian Fondren" ell/dub.sdl:license "MIT" ell/source/ell.d:module ell; ell/source/ell.d:public const int x = 5; ell/source/ell.d:unittest { ell/source/ell.d: assert(x != 5); ell/source/ell.d:} pea/dub.sdl:name "pea" pea/dub.sdl:description "A minimal D application." pea/dub.sdl:authors "Julian Fondren" pea/dub.sdl:copyright "Copyright © 2021, Julian Fondren" pea/dub.sdl:license "MIT" pea/dub.sdl:dependency "ell" version="*" pea/source/pea.d:module pea; pea/source/pea.d:import ell : x; pea/source/pea.d:public const int n = 5; pea/source/pea.d:unittest { pea/source/pea.d: assert(n == x); pea/source/pea.d:} ``` ell has tests that always fail. pea has tests that always succeed. pea depends on ell. ``` $ dub add-local ./ell Registered package: ell (version: ~master) $ cd pea pea$ dub -q test 1 modules passed unittests ``` In this configuration, that ell fails its unittests doesn't matter at all to pea. The library and its value `x` are still used in pea. And ell definitely fails its unittests: ``` pea$ cd ../ell; dub -q test core.exception.AssertError source/ell.d(6): unittest failure ... ... a lot of output ... 1/1 modules FAILED unittests Program exited with code 1 ``` But let's add a single line to ell's dub configuration... ``` ell$ echo 'targetType "sourceLibrary"' >> dub.sdl ell$ cat dub.sdl name "ell" description "A minimal D application." authors "Julian Fondren" copyright "Copyright © 2021, Julian Fondren" license "MIT" targetType "sourceLibrary" ``` ...and we see the behavior that you noted with P and L: ``` ell$ cd ../pea pea$ dub -q test core.exception.AssertError ../ell/source/ell.d(6): unittest failure ... ... a lot of output ... 1/2 modules FAILED unittests ``` length it needs to be. (Meanwhile other problems with dub only does a bunch of git and repository stuff with undesirable slowdowns.)
Sep 08
prev sibling next sibling parent reply Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Tuesday, 31 August 2021 at 16:22:17 UTC, H. S. Teoh wrote:
 On Tue, Aug 31, 2021 at 11:08:46AM +0000, deadalnix via 
 Digitalmars-d wrote:
 On Friday, 27 August 2021 at 14:13:56 UTC, Steven 
 Schveighoffer wrote:
 separate compilation. The compiler doesn't know all the 
 modules that have been built with unittests. Only the 
 runtime does.
 
The is another extremely weird design. When you run the unitests, you want to run the unitests for the module you specifically asked for, not for half the galaxy.
Yeah, this problem has been brought up before, but nobody had the solution. The problem is that when you compile with -unittest, it turns on unittests for ALL modules, including Phobos, 3rd party libraries, *everything*. This is rarely what you want -- Phobos, for example, contains a bunch of extremely heavy duty unittests that end users don't ever want to run. Because of this, the version=StdUnittest hack was implemented in Phobos. But this is not scalable: every library and his neighbour's dog would have to implement their own unittest hack version identifier for this scheme to work. And if some 3rd party library author neglected to do this, you're left out in the cold. I think it makes much more sense to only compile unittests for modules explicitly specified on the command-line. After all, the user is unlikely to be interested in unittesting 3rd party libraries; his primary interest is to unittest his *own* code. However, this simplistic approach falls flat in the face of `dmd -i`. Schemes like `-unittest=mod1,mod2,...` have been proposed before, but AFAIK never implemented. T
Then perhaps have some exlude-unittest argument that accepts packages instead? That would allow to exclude unittest from entire set of modules. Also why do we even make an executable when we compile with -unittest flag? Can't we just make a shared lib, and then have a unit test runner loading this lib and run the tests in it? Imho this would be a lot more flexible. Dmd could come with an default runner, and if someone doesn't like it, he could just replace runner with his own implementation, that would do all the prettyfying he wants. The only thing needed here is well built unit test api exposed to the runner and reflection api at runtime, to allow that said runner inspect underlying tests for additional information, such as annotations, to do more advanced logic than default runner would do. Regards, Alexandru.
Aug 31
parent jfondren <julian.fondren gmail.com> writes:
On Tuesday, 31 August 2021 at 16:52:59 UTC, Alexandru Ermicioi 
wrote:
 Imho this would be a lot more flexible. Dmd could come with an 
 default runner, and if someone doesn't like it, he could just 
 replace runner with his own implementation, that would do all 
 the prettyfying he wants.
This is already the case. Replacing the default runner is what silly's doing here: https://gitlab.com/AntonMeep/silly/-/blob/master/silly.d#L29
 The only thing needed here is well built unit test api exposed 
 to the runner and reflection api at runtime, to allow that said 
 runner inspect underlying tests for additional information, 
 such as annotations, to do more advanced logic than default 
 runner would do.
"at runtime" aside, this is already possible. I give an example of a test runner that runs tests differently depending on its annotations in https://forum.dlang.org/post/bukhjtbxouadyunqwdih forum.dlang.org
Aug 31
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Tuesday, 31 August 2021 at 16:22:17 UTC, H. S. Teoh wrote:
 However, this simplistic approach falls flat in the face of 
 `dmd -i`.  Schemes like `-unittest=mod1,mod2,...` have been 
 proposed before, but AFAIK never implemented.


 T
Why is that? couldn't dmd -i simply not compile the module that are "autoloaded" rather than explicitly provided without unittests?
Aug 31
parent Dennis <dkorpel gmail.com> writes:
On Tuesday, 31 August 2021 at 17:37:34 UTC, deadalnix wrote:
 Why is that? couldn't dmd -i simply not compile the module that 
 are "autoloaded" rather than explicitly provided without 
 unittests?
It could, but then `dmd -i -betterC -unittest -run testrunner.d` would not work anymore, where testrunner.d looks like: ```D module testrunner; static { import moda; import modb; import modc; } alias Seq(T...) = T; extern(C) int main(int argc, const(char)** argv) { foreach(mod; Seq!(moda, modb, modc)) { static foreach (test; __traits(getUnitTests, mod)) { test(); } } return 0; } ```
Aug 31
prev sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
 On Friday, 27 August 2021 at 07:48:40 UTC, Mathias LANG wrote:
 But when I throw in the `-unittest` switch (or the `-cov` or 
 `-profile` for that matter), I want all the bells and 
 whistles. Give me colors, an ncurses interface, an HTTP server 
 even! Give me the full battery pack. Don't expect everyone and 
 their mother to implement their own testing framework, that's 
 just bonkers.
Why would you expect a _compiler_ to implement a testing framework? If the compiler implements the testing framework, any improvement of the testing framework would require an update of the compiler,
It makes for a better default experience, before someone's picked a testing framework and before code's gotten much investment. unittest{} is such a lightweight feature that a single module with a single function may as well have a unittest{} block. Not too long ago dmd was completely silent on a successful -unittest run. Now it says "1 modules passed unittests" at the end. This improved the default experience and no proper testing frameworks were harmed.
 I think it's much better if the testing framework is 
 implementated separate from anything else.
That's already the case. silly, "The better test runner for D", https://code.dlang.org/ and has been in the top five for as long as I can remember. It checks off everyone's complaints in this thread: 1. don't want to care about main? for historical reasons you still have to version(unittest) it out, but as dub is updated that won't be necessary either. 2. want colors? It has red and green unicode symbols. 3. don't want unittests{} blocks cluttering your code? put them in their own modules under tests/ and add "tests/*" to excludedSourceFiles of non-unittest configurations. 4. with minimal configuration following the README.md, all you have to do is run 'dub test' and add tests over time. It's not the case that everyone needs to write their own testing framework. 5. it doesn't bloat the normal build. (if you don't move the dependency on silly into your unittest configuration, the empty module adds 88 bytes to your normal build). silly's another good example of hooking into the runtime's testing support: https://gitlab.com/AntonMeep/silly/-/blob/master/silly.d#L29
 That doesn't mean everybody has to implement it from scratch, 
 just that they should download it from some other location than 
 the compiler website.
 Separation of concerns.

 (by the way, I'm convinced `unittest` is a misfeature of the 
 language. Besides the clear downside that is apparent from this 
 thread,
What clear downside is that? A minor nit about -main? That there's a discussion happening at all? I fixed the betterC bug that I noticed earlier in https://github.com/dlang/dmd/pull/13025 and progress is being made on deadalnix's complaint in https://github.com/dlang/dmd/pull/13026 , and for years people have been running 'dub test' without worrying about this or running 'dmd -unittest -run module', oops, 'dmd -main -unittest -run module', and getting more of the very gentle on-ramp to unit tests than they ever lost to the oops. For the usual internet reasons this discussion is occasionally about 15x as heated as warranted, but since there's no fixing that, you shouldn't sweat it either.
Aug 27
parent reply Johan <j j.nl> writes:
On Friday, 27 August 2021 at 11:15:44 UTC, jfondren wrote:
 On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
 (by the way, I'm convinced `unittest` is a misfeature of the 
 language. Besides the clear downside that is apparent from 
 this thread,
What clear downside is that? A minor nit about -main? That there's a discussion happening at all?
The downside that people expect it to work well. You may be right that the thread is diverging too far from the OP, so I'll stop :) -Johan
Aug 27
parent jfondren <julian.fondren gmail.com> writes:
On Friday, 27 August 2021 at 11:33:14 UTC, Johan wrote:
 On Friday, 27 August 2021 at 11:15:44 UTC, jfondren wrote:
 On Friday, 27 August 2021 at 10:33:02 UTC, Johan wrote:
 (by the way, I'm convinced `unittest` is a misfeature of the 
 language. Besides the clear downside that is apparent from 
 this thread,
What clear downside is that? A minor nit about -main? That there's a discussion happening at all?
The downside that people expect it to work well.
Look at this: ```d /++ dub.sdl: configuration "release" { targetType "executable" } configuration "unittest" { targetType "library" dependency "silly" version="~>1.1.1" } +/ T factorial(T)(T number) { if (number <= 1) return 1; else return number * factorial!T(number - 1); } ("!5") unittest { assert(factorial(5) == 120); } ("!0 and !1") unittest { assert(factorial(0) == 1); assert(factorial(1) == 1); } version (unittest) { } else { void main(string[] args) { import std.conv : to; import std.stdio : writeln; writeln(args[1].to!int.factorial); } } ``` Use it like a script: ``` $ ./fact.d 5; ./fact.d 10 120 3628800 ``` Run its tests (colors lost in translation): ``` $ dub -q test --single fact.d ✓ fact !5 ✓ fact !0 and !1 Summary: 2 passed, 0 failed in 0 ms ``` That's not too bad, is it? It can definitely be improved, but none of my Perl scripts have tests in them. Nevermind the factorial, it's just something I picked out of the pile.
Aug 27
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Aug 24, 2021 at 12:21:41PM +0000, deadalnix via Digitalmars-d wrote:
[...]
 Some of these component are libraries, some are executable. Adding a
 main for libraries is required, or it won't link, but adding one to
 executable is going to cause a link error. This in itself is a major
 main in the ass, because that means there is no one consistent way to
 unittest a module without knowing if that module has a main. In
 addition, everything needs to be built twice now, which is really
 undesirable.
There's the `-main` flag for adding an empty main() to the compiled sources. But yeah, if there's already a main() it will die with a compile error. The hack solution for this is to prepend this to your main(), like this: version(unittest){} else void main() { ... } The point about building everything twice is legit, and is a pain point I myself have come across. In the old days, -unittest used to just run unittests before invoking main(). Then somebody changed this so that -unittest causes main() not to run. So when coding/compiling/debugging, I now need to build the program twice: once with -unittest, once without. Technically this isn't avoidable (no) thanks to `version(unittest)`: the object files may have very different contents so compiling twice is almost unavoidable. But in the normal, common case, there isn't *that* much of a difference, and it would be nice to only have to compile once (like in the old days).
 Maybe one could rdmd each modules to runt he tests? That seems like
 the best approach to me, considering one doesn't want to actually
 produce an artifact for the tests, simply run them. this also has the
 main/no main problem, but in addition, it often fails with
 incomprehensible linker errors. For some reason, rdmd is not able to
 include all dependencies consistently, but, even better, it doesn't
 take the same standard flags as other linkers do, so it is not
 possible to reuse existing build infrastructure to feed all the
 correct flags to rdmd.
Honestly, these days I find little need for rdmd. I can just run `dmd -unittest -main -i -run main.d` and it will compile unittests, insert an empty main(), and pull in all imports, run the resulting executable, and clean up afterwards. Just a single command line takes care of it all. Well, modulo the `version(unittest) else` hack for main().
 This may seems like it wouldn't be that big of a deal if you manage
 all your flags by yourself, but very quickly turns into a nightmare
 once you have to use 3rd party libraries that ship with their own set
 of flags.
Which means we need dub support for this. :-D
 At this point, I would just wish that one could rdmd --unittest
 modulename.d and just pass it a couple of -I, -L and -l flags and have
 all of it work. I have no idea how to achieve that.
[...] As I already said: dmd -unittest -main -i -run main.d T -- Only boring people get bored. -- JM
Aug 24
parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 24 August 2021 at 17:03:36 UTC, H. S. Teoh wrote:
 As I already said:

 	dmd -unittest -main -i -run main.d


 T
I was not aware of this feature. This is pretty good and a good step forward. thanks for letting me know.
Aug 25
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/24/2021 5:21 AM, deadalnix wrote:
 Adding a main for libraries is required, or it won't link, but adding one to 
 executable is going to cause a link error.
This is indeed a real problem. It was solved with the -main switch to dmd, which will just stick one in for you. I use it all the time for unit testing modules.
Aug 25
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 25 August 2021 at 20:49:32 UTC, Walter Bright wrote:
 On 8/24/2021 5:21 AM, deadalnix wrote:
 Adding a main for libraries is required, or it won't link, but 
 adding one to executable is going to cause a link error.
This is indeed a real problem. It was solved with the -main switch to dmd, which will just stick one in for you. I use it all the time for unit testing modules.
The problem with -main is that it's too blunt an instrument: it will add a main function whether the code you're compiling already has one or not. That means you have to keep track of which modules have main functions and which ones don't, so that you (or more realistically, your build system) can add -main only when it's necessary. If we had something like -main=ifMissing that added an empty main function only if the modules being compiled didn't already have one, then it would be possible to unit-test any D module with a command line like dmd -i -unittest -main=ifMissing -run mymodule.d Being able to use the same command for everything makes automation much easier--you can write it in a makefile rule, put it in a script, etc.
Aug 25
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Wednesday, 25 August 2021 at 21:05:42 UTC, Paul Backus wrote:
 If we had something like -main=ifMissing that added an empty 
 main function only if the modules being compiled didn't already 
 have one, then it would be possible to unit-test any D module 
 with a command line like

     dmd -i -unittest -main=ifMissing -run mymodule.d

 Being able to use the same command for everything makes 
 automation much easier--you can write it in a makefile rule, 
 put it in a script, etc.
Although at this point, the -main=ifMissing functionality should just be part of -unittest
Aug 25
parent reply jfondren <julian.fondren gmail.com> writes:
On Wednesday, 25 August 2021 at 21:17:54 UTC, jfondren wrote:
 On Wednesday, 25 August 2021 at 21:05:42 UTC, Paul Backus wrote:
 If we had something like -main=ifMissing that added an empty 
 main function only if the modules being compiled didn't 
 already have one, then it would be possible to unit-test any D 
 module with a command line like

     dmd -i -unittest -main=ifMissing -run mymodule.d

 Being able to use the same command for everything makes 
 automation much easier--you can write it in a makefile rule, 
 put it in a script, etc.
Although at this point, the -main=ifMissing functionality should just be part of -unittest
Or rather, -unittest -run ? Since you could dmd -c -unittest a bunch of modules and then link them together.
Aug 25
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 25 August 2021 at 21:19:36 UTC, jfondren wrote:
 On Wednesday, 25 August 2021 at 21:17:54 UTC, jfondren wrote:
 On Wednesday, 25 August 2021 at 21:05:42 UTC, Paul Backus 
 wrote:
 If we had something like -main=ifMissing that added an empty 
 main function only if the modules being compiled didn't 
 already have one, then it would be possible to unit-test any 
 D module with a command line like

     dmd -i -unittest -main=ifMissing -run mymodule.d

 Being able to use the same command for everything makes 
 automation much easier--you can write it in a makefile rule, 
 put it in a script, etc.
Although at this point, the -main=ifMissing functionality should just be part of -unittest
Or rather, -unittest -run ? Since you could dmd -c -unittest a bunch of modules and then link them together.
Even if we have a rule like "-unittest -run implies -main=ifMissing", I think there should still be a separate flag for it--for documentation if nothing else.
Aug 25
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/25/2021 2:46 PM, Paul Backus wrote:
     dmd -i -unittest -main=ifMissing -run mymodule.d
The trouble with adding switches like that is it takes a while to understand them and what they do, and they'll inevitably create yet another problem which will require another switch. It's better to use one of the existing methods shown in this thread which use already well-understood behavior.
Aug 25
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/25/2021 2:05 PM, Paul Backus wrote:
 The problem with -main is that it's too blunt an instrument: it will add a
main 
 function whether the code you're compiling already has one or not. That means 
 you have to keep track of which modules have main functions and which ones 
 don't, so that you (or more realistically, your build system) can add -main
only 
 when it's necessary.
That problem has never occurred to me, and has never happened to me, yet I use -main all the time. One way to resolve it is just put your main() in a separate module.
Aug 25
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Aug 25, 2021 at 03:44:40PM -0700, Walter Bright via Digitalmars-d wrote:
 On 8/25/2021 2:05 PM, Paul Backus wrote:
 The problem with -main is that it's too blunt an instrument: it will
 add a main function whether the code you're compiling already has
 one or not. That means you have to keep track of which modules have
 main functions and which ones don't, so that you (or more
 realistically, your build system) can add -main only when it's
 necessary.
That problem has never occurred to me, and has never happened to me, yet I use -main all the time. One way to resolve it is just put your main() in a separate module.
This always happens to me, so much so that I've come to insert the following as boilerplate to all my code: version(unittest){} else int main(string[] args) { ... } T -- LINUX = Lousy Interface for Nefarious Unix Xenophobes.
Aug 25
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/25/2021 3:48 PM, H. S. Teoh wrote:
 This always happens to me, so much so that I've come to insert the
 following as boilerplate to all my code:
 
 	version(unittest){} else
 	int main(string[] args) { ... }
That works, too.
Aug 25
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 25 August 2021 at 22:44:40 UTC, Walter Bright wrote:
 On 8/25/2021 2:05 PM, Paul Backus wrote:
 The problem with -main is that it's too blunt an instrument: 
 it will add a main function whether the code you're compiling 
 already has one or not. That means you have to keep track of 
 which modules have main functions and which ones don't, so 
 that you (or more realistically, your build system) can add 
 -main only when it's necessary.
That problem has never occurred to me, and has never happened to me, yet I use -main all the time. One way to resolve it is just put your main() in a separate module.
What if you are dealing with something that is a single file? If you run below on run.dlang.org with either `-unittest` or `-unittest -main`, then you get a message that only one main is allowed. If you remove the dependency, then it compiles without error with `-unittest` but gets the same error with `-unittest -main`. The difference is (IIRC) when you include a dependency it is also calling `-single`. If you had something like `-main=ifmissing` that others describe, then there would be a consistent solution that could work well with run.dlang.org. ```d /+dub.sdl: dependency "mir-algorithm" versio="*" +/ unittest {} void main() {} ```
Aug 25
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Wednesday, 25 August 2021 at 20:49:32 UTC, Walter Bright wrote:
 On 8/24/2021 5:21 AM, deadalnix wrote:
 Adding a main for libraries is required, or it won't link, but 
 adding one to executable is going to cause a link error.
This is indeed a real problem. It was solved with the -main switch to dmd, which will just stick one in for you. I use it all the time for unit testing modules.
No it wasn't, because it add a main no matter what, which means that when there is a main in the module being tested, you get a link error. The root of the problem, really, is that I don't cares about main when running unit tests. I don't want to know if it is there or not, I don't want it to run, nothing. If the compiler needs one to link under the hood, then good for it, but this is an implementation detail. When running the unit tests, one wants to know if the unit tests pass. Nothing more, nothing less. The compiler has 100% of the information it needs to make this happen, yet, for some reason, it is requesting them from me. Which i can provide too, but this is fragile, and it doesn't scale, as I can't get my build system to automate it for me. Which in turn lead to most projects rolling out their own solution. I can work with having to pass the -main flag. This is not ideal, because what else could I possibly want? A link error? I'm not trying to link and run main. I care about unittests in that case. But having main declared twice is some special level of nonsense. In what case could I possibly want two main?
Aug 26
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Thursday, 26 August 2021 at 09:38:54 UTC, deadalnix wrote:
 The root of the problem, really, is that I don't cares about 
 main when running unit tests. I don't want to know if it is 
 there or not, I don't want it to run, nothing. If the compiler 
 needs one to link under the hood, then good for it, but this is 
 an implementation detail.
OK. Create a bugzilla issue. There are some related bugs that and -of). This is about the extent of the current support, in src/dmd/mars.d's tryMain: ```d if (params.addMain) files.push("__main.d"); // Create Modules Modules modules = createModules(files, libmodules); // Read files // Start by "reading" the special files (__main.d, __stdin.d) foreach (m; modules) { if (params.addMain && m.srcfile.toString() == "__main.d") { auto data = arraydup("int main(){return 0;}\0\0\0\0"); // need 2 trailing nulls for sentinel and 2 for lexer m.srcBuffer = new FileBuffer(cast(ubyte[]) data[0 .. $-4]); } ``` and src/dmd/glue.d: ```d private bool onlyOneMain(Loc loc) { __gshared Loc lastLoc; __gshared bool hasMain = false; if (hasMain) { const(char)* msg = ""; if (global.params.addMain) msg = ", -main switch added another `main()`"; ``` It's not surprising that such a lightweight implementation doesn't result in world-class ergonomics. It's something I use all the time to test individual modules at the CLI, and I'd much rather have this than nothing. But for bells and whistles I'd look to dub. In a pinch you could write a dmd wrapper that calls dmd, sees if it gets a linker complaint about main, and if so calls dmd again with -main. An obvious performance hit, probably not that bad in most cases where you would consider this, but now you don't have care about main. ```d import std.process : execute, spawnProcess, wait; import std.algorithm : canFind; import std.stdio : write; int main(string[] args) { auto result = execute(["dmd"] ~ args[1..$]); if (result.status != 0 && result.output.canFind("undefined reference to `main'")) { return spawnProcess(["dmd", "-main"] ~ args[1..$]).wait; } write(result.output); return result.status; } ``` usage: ``` $ cat nomain.d unittest { import std.stdio : writeln; writeln("no main"); } $ cat hasmain.d unittest { import std.stdio : writeln; writeln("has main"); } void main() {} $ ./automain.d -unittest -run nomain.d no main 1 modules passed unittests $ ./automain.d -unittest -run hasmain.d 1 modules passed unittests has main ```
Aug 26
parent deadalnix <deadalnix gmail.com> writes:
On Thursday, 26 August 2021 at 11:32:06 UTC, jfondren wrote:
 On Thursday, 26 August 2021 at 09:38:54 UTC, deadalnix wrote:
 The root of the problem, really, is that I don't cares about 
 main when running unit tests. I don't want to know if it is 
 there or not, I don't want it to run, nothing. If the compiler 
 needs one to link under the hood, then good for it, but this 
 is an implementation detail.
OK. Create a bugzilla issue. There are some related bugs that with -c and -of). This is about the extent of the current support, in src/dmd/mars.d's tryMain: ```d if (params.addMain) files.push("__main.d"); // Create Modules Modules modules = createModules(files, libmodules); // Read files // Start by "reading" the special files (__main.d, __stdin.d) foreach (m; modules) { if (params.addMain && m.srcfile.toString() == "__main.d") { auto data = arraydup("int main(){return 0;}\0\0\0\0"); // need 2 trailing nulls for sentinel and 2 for lexer m.srcBuffer = new FileBuffer(cast(ubyte[]) data[0 .. $-4]); } ``` and src/dmd/glue.d: ```d private bool onlyOneMain(Loc loc) { __gshared Loc lastLoc; __gshared bool hasMain = false; if (hasMain) { const(char)* msg = ""; if (global.params.addMain) msg = ", -main switch added another `main()`"; ``` It's not surprising that such a lightweight implementation doesn't result in world-class ergonomics. It's something I use all the time to test individual modules at the CLI, and I'd much rather have this than nothing. But for bells and whistles I'd look to dub. In a pinch you could write a dmd wrapper that calls dmd, sees if it gets a linker complaint about main, and if so calls dmd again with -main. An obvious performance hit, probably not that bad in most cases where you would consider this, but now you don't have care about main. [...]
Congratulations, you just wrote the embryo of a test framework. It will work, and indeed, most projects ou there get one past a certain size. It' very unfortunate though, because the compiler has all the info it needs to do the right thing, and yet refuses to do it. Note that you could likely get better perfs out of this by linking in a weak definition of _Dmain, so it won't clash, but the general problem remains.
Aug 26
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Thursday, 26 August 2021 at 09:38:54 UTC, deadalnix wrote:
 But having main declared twice is some special level of 
 nonsense. In what case could I possibly want two main?
When there's both a main function and the `-main` flag, should `-main` do nothing or override the main function with an empty one?
Aug 26
next sibling parent jfondren <julian.fondren gmail.com> writes:
On Thursday, 26 August 2021 at 23:29:20 UTC, Dennis wrote:
 On Thursday, 26 August 2021 at 09:38:54 UTC, deadalnix wrote:
 But having main declared twice is some special level of 
 nonsense. In what case could I possibly want two main?
When there's both a main function and the `-main` flag, should `-main` do nothing or override the main function with an empty one?
In the -unittest usage, and without some code overriding the runtime's testrunner, these are equivalent outcomes since dmd no longer runs main after unittests: ```d import std.stdio; unittest { writeln("unittest"); } void main() { writeln("main"); } ``` contrast: ``` $ dmd -unittest -run ignoremain.d unittest 1 modules passed unittests $ gdc -funittest -o not-ignored ignoremain.d $ ./not-ignored unittest main ```
Aug 26
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 26 August 2021 at 23:29:20 UTC, Dennis wrote:
 On Thursday, 26 August 2021 at 09:38:54 UTC, deadalnix wrote:
 But having main declared twice is some special level of 
 nonsense. In what case could I possibly want two main?
When there's both a main function and the `-main` flag, should `-main` do nothing or override the main function with an empty one?
This is the wrong question. When one runs the unittests, they don't care about main. The existance of the main flag is dubious to begin with.
Aug 27
parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 27 August 2021 at 10:30:58 UTC, deadalnix wrote:
 This is the wrong question. When one runs the unittests, they 
 don't care about main. The existance of the main flag is 
 dubious to begin with.
Saying "-unittest should have been designed differently" doesn't help us forward. Currently it's possible to have a -betterC test runner in main, or set `UnitTestResult.runmain = true` from `core.runtime`, or to create a unittest build with incremental separate compilation, so changing the semantics of the -unittest flag is a very disruptive change. However, like you said, there is no use case for having two `main` functions in one build, so I was looking into improving that (see https://github.com/dlang/dmd/pull/13026), which is how I got to that very real implementation question. jfondren pointed out that the default test runner doesn't run main, so I think it's best to make -main only insert main when there isn't already one being compiled in. Then you should be able to do `dmd -unittest -main -run somemodule.d` on any (self-contained) d file with consistent results.
Aug 27
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/27/21 9:35 AM, Dennis wrote:
 On Friday, 27 August 2021 at 10:30:58 UTC, deadalnix wrote:
 This is the wrong question. When one runs the unittests, they don't 
 care about main. The existance of the main flag is dubious to begin with.
Saying "-unittest should have been designed differently" doesn't help us forward. Currently it's possible to have a -betterC test runner in main, or set `UnitTestResult.runmain = true` from `core.runtime`, or to create a unittest build with incremental separate compilation, so changing the semantics of the -unittest flag is a very disruptive change. However, like you said, there is no use case for having two `main` functions in one build, so I was looking into improving that (see https://github.com/dlang/dmd/pull/13026), which is how I got to that very real implementation question. jfondren pointed out that the default test runner doesn't run main, so I think it's best to make -main only insert main when there isn't already one being compiled in. Then you should be able to do `dmd -unittest -main -run somemodule.d` on any (self-contained) d file with consistent results.
The default test runner doesn't run main *by default*, but it also *can* run main if passed a runtime option. So it still needs a main to run if requested. I think a "-nomain" option is something worth looking at (basically, the generated `extern(C) main` function doesn't call a missing `dmain` function). It can't be tied to `-unittest` though. -Steve
Aug 27
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/26/2021 2:38 AM, deadalnix wrote:
 I can work with having to pass the -main flag. This is not ideal, because what 
 else could I possibly want?
Then we'll get a bug report where the user compiled a module, and it compiled and linked without error, then he runs the program and nothing happens because the compiler inserted an empty main().
Aug 27
parent reply deadalnix <deadalnix gmail.com> writes:
On Saturday, 28 August 2021 at 00:47:44 UTC, Walter Bright wrote:
 On 8/26/2021 2:38 AM, deadalnix wrote:
 I can work with having to pass the -main flag. This is not 
 ideal, because what else could I possibly want?
Then we'll get a bug report where the user compiled a module, and it compiled and linked without error, then he runs the program and nothing happens because the compiler inserted an empty main().
No, because it'll run the unittests, and the user will see "X test passed" or something similar when running the executable, understand that they ran the uni tests and go ahead and run the executable that has been built without the unitests flag. If the user doesn't have any unitests in the executable, then saying "0 test run, 0 passed, 0 failed" is enough to indicated to the user that this was a unitests build.
Aug 31
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/31/2021 4:04 AM, deadalnix wrote:
 No, because it'll run the unittests, and the user will see "X test passed" or 
 something similar when running the executable, understand that they ran the
uni 
 tests and go ahead and run the executable that has been built without the 
 unitests flag.
 
 If the user doesn't have any unitests in the executable, then saying "0 test 
 run, 0 passed, 0 failed" is enough to indicated to the user that this was a 
 unitests build.
The unittest feature doesn't do any of those things unless you code it to.
Sep 05
parent reply deadalnix <deadalnix gmail.com> writes:
On Sunday, 5 September 2021 at 21:00:15 UTC, Walter Bright wrote:
 On 8/31/2021 4:04 AM, deadalnix wrote:
 No, because it'll run the unittests, and the user will see "X 
 test passed" or something similar when running the executable, 
 understand that they ran the uni tests and go ahead and run 
 the executable that has been built without the unitests flag.
 
 If the user doesn't have any unitests in the executable, then 
 saying "0 test run, 0 passed, 0 failed" is enough to indicated 
 to the user that this was a unitests build.
The unittest feature doesn't do any of those things unless you code it to.
It already output a very similar message: "15 modules passed unittests" The specifics of the message do not matter here.
Sep 05
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/5/2021 3:00 PM, deadalnix wrote:
 On Sunday, 5 September 2021 at 21:00:15 UTC, Walter Bright wrote:
 The unittest feature doesn't do any of those things unless you code it to.
It already output a very similar message: "15 modules passed unittests"
That must be a fairly recent change I wasn't aware of.
Sep 06
parent reply jfondren <julian.fondren gmail.com> writes:
On Monday, 6 September 2021 at 16:35:03 UTC, Walter Bright wrote:
 On 9/5/2021 3:00 PM, deadalnix wrote:
 On Sunday, 5 September 2021 at 21:00:15 UTC, Walter Bright 
 wrote:
 The unittest feature doesn't do any of those things unless 
 you code it to.
It already output a very similar message: "15 modules passed unittests"
That must be a fairly recent change I wasn't aware of.
It came in with https://github.com/dlang/druntime/pull/1685 , which also added a lot more control over how the tests could be run. Old behavior: adding -unittest causes unittest functions to be compiled in and to run before main, and that's it: the program only exited before main on error because that's how assert() failures are handled, and the program continued to run if there were no errors. This perfectly fits the use case of "I want a build of my program with a bunch of extra testing, ala a debug or a non-release build" but required some workarounds to fit the use case of "I want to run all of my unittests, *only*" Much of this thread can be listed under complaints about a third use case, "I want to run a single module's tests, *only*"
Sep 06
parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 6 September 2021 at 16:52:38 UTC, jfondren wrote:
 It came in with https://github.com/dlang/druntime/pull/1685 , 
 which also added a lot more control over how the tests could be 
 run.
https://github.com/dlang/druntime/pull/1685#issuecomment-256795118 Damn, I don't know who that guy is, but it seems like he foresaw the current mess that we are in.
Sep 06
next sibling parent reply jfondren <julian.fondren gmail.com> writes:
On Monday, 6 September 2021 at 16:55:30 UTC, deadalnix wrote:
 On Monday, 6 September 2021 at 16:52:38 UTC, jfondren wrote:
 It came in with https://github.com/dlang/druntime/pull/1685 , 
 which also added a lot more control over how the tests could 
 be run.
https://github.com/dlang/druntime/pull/1685#issuecomment-256795118 Damn, I don't know who that guy is, but it seems like he foresaw the current mess that we are in.
-main wasn't and still isn't a problem for the use case the PR was authored for, of "I want to run all my unit tests, *only*", and the approach of the PR approach was completely successful at making druntime much more friendly to that use case. It had zero impact on your use case, but you didn't sell it very well by only mentioning what for other uses was a very trivial inconvenience.
Sep 06
parent reply jfondren <julian.fondren gmail.com> writes:
On Monday, 6 September 2021 at 22:53:50 UTC, deadalnix wrote:
 On Monday, 6 September 2021 at 17:12:50 UTC, jfondren wrote:
 -main wasn't and still isn't a problem for the use case the PR 
 was authored for, of "I want to run all my unit tests, 
 *only*", and the approach of the PR approach was completely 
 successful at making druntime much more friendly to that use 
 case. It had zero impact on your use case, but you didn't sell 
 it very well by only mentioning what for other uses was a very 
 trivial inconvenience.
[something deadalnix typed a while ago and has been waiting to post]
The blindness you have to precisely how people intend to use features, "use case blindness", has resulted in a lot of trouble for you: 1. you earn pointless animosity by describing existing features, which other people use all the time in comfort, as entirely broken and obviously badly designed. 2. you keep derailing your own thread about your own use case, because you don't think it's important to keep the focus on *your* use case. 3. even when you talk about problems that directly relate to pretty severe inconveniences for your use case, you're unable to interest other people in them, because you don't think to connect the problems to a use case where they *are* severe inconveniences. You keep appealing to experiences that are actually personal to you and not generally felt. I've mostly enjoyed this thread and appreciate that you made it; I dug a lot into unit tests in D as a result of following it. The author of the PR that you think has helped 'lead to this mess' has said he's happy fine with the state of D unit testing. Just like other people are not all constantly slapping their foreheads over how irritating -main is to use, and just like the other participants of that PR didn't think "it doesn't fix -main" was a showstopper for that PR, other people also do not all think that "General has a 80 post thread about unit tests" is an argument in itself than unit tests have a problem. I think a still likely outcome is that a general interest in unittests will result in a lot of improvements, but only to other people's use cases. If that's the case in a few months, give Discord a try.
Sep 06
parent Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Monday, 6 September 2021 at 23:57:05 UTC, jfondren wrote:
 The blindness you have to precisely how people intend to use 
 features, "use case blindness", has resulted in a lot of 
 trouble for you:

 1. you earn pointless animosity by describing existing 
 features, which other people use all the time in comfort, as 
 entirely broken and obviously badly designed.
Doesn't mean that it should not be improved, or properly fixed. Oh and if someone doesn't complain about them, doesn't mean that they have good architecture.
 2. you keep derailing your own thread about your own use case, 
 because you don't think it's important to keep the focus on 
 *your* use case.
Selecting just a few modules to compile in a large project will affect not only him, but all projects that use source libraries that have long compile time of the unit tests.
 3. even when you talk about problems that directly relate to 
 pretty severe inconveniences for your use case, you're unable 
 to interest other people in them, because you don't think to 
 connect the problems to a use case where they *are* severe 
 inconveniences.
Why are you trying to infer what he thinks, and post this as his thoughts? He did interest at least me, since I had experience of waiting unit tests compile for a source library that I didn't need to in first place. And to tell the truth it wasn't a nice experience, given D advertises fast compilation cycles. Best regards, Alexandru.
Sep 07
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 6 September 2021 at 16:55:30 UTC, deadalnix wrote:
 Damn, I don't know who that guy is, but it seems like he 
 foresaw the current mess that we are in.
The comment reads:
 I'm not convinced this is the right approach. The thing will 
 still fail to link when no main function is provided.

 IMO, it is better to do this as proposed by basil. I plan to do 
 a DMD PR.
So what happened to that plan? What's basil's proposal?
Sep 06
parent deadalnix <deadalnix gmail.com> writes:
On Monday, 6 September 2021 at 20:19:02 UTC, Dennis wrote:
 I'm not convinced this is the right approach. The thing will 
 still fail to link when no main function is provided.

 IMO, it is better to do this as proposed by basil. I plan to 
 do a DMD PR.
So what happened to that plan? What's basil's proposal?
I have no idea, I do not even remember having posted that comment.
Sep 06
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Thursday, 26 August 2021 at 09:38:54 UTC, deadalnix wrote:
 On Wednesday, 25 August 2021 at 20:49:32 UTC, Walter Bright 
 wrote:
 This is indeed a real problem. It was solved with the -main 
 switch to dmd, which will just stick one in for you. I use it 
 all the time for unit testing modules.
No it wasn't, because it add a main no matter what, which means that when there is a main in the module being tested, you get a link error.
Good news: this is now fixed! https://github.com/dlang/dmd/pull/13057 Grab a [nightly](https://github.com/dlang/dmd/releases/tag/nightly) and try out: `dmd -unittest -main -run`
Sep 11
parent deadalnix <deadalnix gmail.com> writes:
On Saturday, 11 September 2021 at 21:23:07 UTC, Dennis wrote:
 Good news: this is now fixed!

 https://github.com/dlang/dmd/pull/13057

 Grab a 
 [nightly](https://github.com/dlang/dmd/releases/tag/nightly) 
 and try out: `dmd -unittest -main -run`
https://github.com/dlang/dmd/pull/13057#issuecomment-918061875
Sep 13