www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Should unittests run as logical part of compilation?

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
There's this simple realization that unittests could (should?) be 
considered an intrinsic part of the build process. In order for an 
executable to be worth running, it should pass the regular semantic 
checks and also the unittests, which in a sense are extended semantic 
checks that fall outside the traditional charter of the compiler.

In that view, asserts inside unittests should fail with the same message 
format as regular compilation errors, i.e.

./modulename.d(103): Unittest failed: user-defined message

Stack traces and other artifacts printed by failing asserts should come 
after, indented. An IDE user should run unittest with the usual "build" 
command and be able to click on the unittest failures just the same as 
build errors.

In particular, this view of unittests declares our current stance on 
running unittests ("run unittests just before main()") as meaningless. 
Indeed that has bothered me for quite a while - unittests are part of 
the build/acceptance, not part of every run. To wit, this is a growing 
idiom in D programs:

version(unittest) void main() {}
else void main()
{
     ...
}

What do you think? Logistically it shouldn't be too hard to arrange 
things to cater to this approach.


Andrei
Jan 25 2014
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei

+1, unittest passed as command line argument to dmd should run unittests as compilation step and main should remain untouched.
Jan 25 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

This idiom is probably mostly obsolete now that we have the -main flag. An IDE can build with -unittest -main -run and exclude the source file that normally defines `main`.
 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.

I like it, as long as it doesn't mean unit tests are enabled by default, as that would probably be very surprising both for building release binaries and from a security perspective.
Jan 25 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 25 Jan 2014 14:55:30 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 There's this simple realization that unittests could (should?) be 
 considered an intrinsic part of the build process. In order for an 
 executable to be worth running, it should pass the regular semantic 
 checks and also the unittests, which in a sense are extended semantic 
 checks that fall outside the traditional charter of the compiler.
 
 [....]
 
 Andrei

But please keep some way to manually run unittests - otherwise cross compilers will have lots of fun running unittests ;-) (IMHO this is the responsibility of a build tool, not of the compiler)
Jan 25 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

This is an example of bit a more general need: in D I'd like to create modules that can be imported from other modules to use their functionality; and to run their own main() when they are not imported from other modules. It's a simple feature that is still missing in D. Bye, bearophile
Jan 25 2014
prev sibling next sibling parent reply "Mike" <none none.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.
 [...]
 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.

How does this work for those who build on one machine and test on another? I'm currently building on a Linux host, and deploying to a bare-metal ARM Cortex-M processor. Also, with limited amounts of flash, I may only have enough room for unit tests or my program, but not both. (Just so you know, I don't actually use the unit test feature yet, but I intend to) This seems more like a workflow feature rather than a build feature. But integration into the toolchain could be pretty cool (like you said, clicking on a messages in the output window, and zooming right to the point of failure). It seems to me that having an additional tool in the tool chain (the unit tester) along with the compiler, linker, debugger, etc.. that could be initiated by the build process, is the way to go. If compiling unit tests, the build process could generate a unit test executable, run the unit test executable, and, assuming all tests pass, then generate the production executable. I think such a tool could scale to my platform, somehow, I suppose. Mike
Jan 25 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 4:26 PM, Mike wrote:
 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.
 [...]
 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.

How does this work for those who build on one machine and test on another?

Yah, thanks (and Johannes too) for making this point. That means we should not prevent the option of running them separately.
 This seems more like a workflow feature rather than a build feature.
 But integration into the toolchain could be pretty cool (like you said,
 clicking on a messages in the output window, and zooming right to the
 point of failure).

That is correct - an build environment thing. Andrei
Jan 26 2014
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

I already attempt to get unittests to fail at compile time by using static assert. I think running it during compilation would be a great change. Though cross compiling is a valid concern and should be addressed. Many are use to the notion that unittest are separate from compilation, I think this is only due to familiarity. Other than the cross-compile issue, I believe moving it closer to the time of compilation is the correct direction, but it will get distaste from those familiar with JUnit/NUnit and other unittesting frameworks just as it currently does.
Jan 25 2014
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-26 04:52, Jesse Phillips wrote:

 I already attempt to get unittests to fail at compile time by using
 static assert. I think running it during compilation would be a great
 change. Though cross compiling is a valid concern and should be addressed.

It's already possible to do CTFE unit tests: http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com -- /Jacob Carlborg
Jan 26 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 7:52 PM, Jesse Phillips wrote:
 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the same
 message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

I already attempt to get unittests to fail at compile time by using static assert. I think running it during compilation would be a great change.

That's great but I wasn't thinking of CTFEing unittests as much as just running them right after building. Andrei
Jan 26 2014
prev sibling next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 ./modulename.d(103): Unittest failed: user-defined message

be invaluable to doing this without adding this specific behaviour. (Really should deal with that.)
 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

runtime would be useful. Because what happens when you need to test e.g. an OS feature with it? Or have a dependency that simply cannot run at compile time? I'm all for being able to selectively run unittests and having the ability to have some run at compile time.
Jan 25 2014
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-26 05:29, Rikki Cattermole wrote:

 I'm all for being able to selectively run unittests and having the
 ability to have some run at compile time.

That's already possible: http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com -- /Jacob Carlborg
Jan 26 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 8:29 PM, Rikki Cattermole wrote:
 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
 ./modulename.d(103): Unittest failed: user-defined message

invaluable to doing this without adding this specific behaviour. (Really should deal with that.)

What about static assert(false, "message")? Andrei
Jan 26 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 3:20 AM, anonymous wrote:
 On Sunday, 26 January 2014 at 05:17:15 UTC, H. S. Teoh wrote:
 On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole wrote:
 Having the ability to run unittests at both compile time and runtime
 would be useful. Because what happens when you need to test e.g. an
 OS feature with it? Or have a dependency that simply cannot run at
 compile time?

 I'm all for being able to selectively run unittests and having the
 ability to have some run at compile time.

+1. Some of my unittests can only be run at runtime: like testing file I/O. Obviously, other unittests can also be run at compile-time, so it's useful to have both.

I don't think the tests are supposed to be CTFE'd. They'd be compiled (to a temporary executable?) and then run normally.

Yah, that was my idea. Andrei
Jan 26 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole wrote:
 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu
 wrote:

In particular, this view of unittests declares our current stance
on running unittests ("run unittests just before main()") as
meaningless. Indeed that has bothered me for quite a while -
unittests are part of the build/acceptance, not part of every run.
To wit, this is a growing idiom in D programs:

Having the ability to run unittests at both compile time and runtime would be useful. Because what happens when you need to test e.g. an OS feature with it? Or have a dependency that simply cannot run at compile time? I'm all for being able to selectively run unittests and having the ability to have some run at compile time.

+1. Some of my unittests can only be run at runtime: like testing file I/O. Obviously, other unittests can also be run at compile-time, so it's useful to have both. T -- Кто везде - тот нигде.
Jan 25 2014
prev sibling next sibling parent "anonymous" <anonymous example.com> writes:
On Sunday, 26 January 2014 at 05:17:15 UTC, H. S. Teoh wrote:
 On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole 
 wrote:
 Having the ability to run unittests at both compile time and 
 runtime
 would be useful. Because what happens when you need to test 
 e.g. an
 OS feature with it? Or have a dependency that simply cannot 
 run at
 compile time?
 
 I'm all for being able to selectively run unittests and having 
 the
 ability to have some run at compile time.

+1. Some of my unittests can only be run at runtime: like testing file I/O. Obviously, other unittests can also be run at compile-time, so it's useful to have both.

I don't think the tests are supposed to be CTFE'd. They'd be compiled (to a temporary executable?) and then run normally.
Jan 26 2014
prev sibling next sibling parent reply "simendsjo" <simendsjo gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless.

I wouldn't mind having unittests be a part of the compilation process, but I really don't think "running before main" is useless. This lets me compile a version on my desktop and push the binary to a small VPS staging for running runtime tests. Compiling any non-trivial code on 512MB with the DMD frontend is impossible. I can also send a fully working application with unittests to a client as a beta before giving a production version.
Jan 26 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 3:25 AM, simendsjo wrote:
 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process.

 In particular, this view of unittests declares our current stance on
 running unittests ("run unittests just before main()") as meaningless.

I wouldn't mind having unittests be a part of the compilation process, but I really don't think "running before main" is useless. This lets me compile a version on my desktop and push the binary to a small VPS staging for running runtime tests. Compiling any non-trivial code on 512MB with the DMD frontend is impossible. I can also send a fully working application with unittests to a client as a beta before giving a production version.

Does it help that the client runs them every time? I.e. does the n+1th run of the unittests bring more information than the first? Andrei
Jan 26 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 26 January 2014 at 11:25:25 UTC, simendsjo wrote:
 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei 
 Alexandrescu wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless.

I wouldn't mind having unittests be a part of the compilation process, but I really don't think "running before main" is useless. This lets me compile a version on my desktop and push the binary to a small VPS staging for running runtime tests. Compiling any non-trivial code on 512MB with the DMD frontend is impossible. I can also send a fully working application with unittests to a client as a beta before giving a production version.

I simply think that "-unittest" should not emit any existing "main" function by default (only stub one if --main is set) and other than that we are pretty good at current stage. Running tests is necessary part of build system indeed but dmd is not a build system and should not force any assumptions about it. Initial idea to make tests always run before the program starts does not work though, that is true. But currently we have enough tools to do it in any way specific project needs.
Jan 26 2014
prev sibling next sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei

I don't think unit tests should be run on *each* build - if I'm fidgeting with a specific piece of my code, and building&running it after each change, I don't want to waste time running the unittests of the entire project. I do agree that unittests shouldn't be part of each run though. I think the `-unittest` flag should ignore the `main` function altogether, creating a unittests only executable, and the build system will be responsible for running it. How about keeping the old behavior of `-unittest` for backward compatibility, and having a new syntax - `-unittest=filename`, where "filename" is the name of the unittests executable. That way both the unittest and the regular executables will be created at the same time - which will speed things up, as most of the compilation work can be shared between those two builds - and have different file names, so the build system - which will make configuring the build system more straightforward. If we make the unittest a separate executable, there won't be a `main` function waiting for command line arguments, so we can use the command line arguments for the unittests. We can have flags for only running tests of specific packages, or for running the tests under certain restrictions etc.
Jan 26 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 4:58 AM, Idan Arye wrote:
 I don't think unit tests should be run on *each* build - if I'm
 fidgeting with a specific piece of my code, and building&running it
 after each change, I don't want to waste time running the unittests of
 the entire project.

Hah, interesting. So unittests of one module should run again iff there were changes in that module or other modules that that depends on.
 How about keeping the old behavior of `-unittest` for backward
 compatibility, and having a new syntax - `-unittest=filename`, where
 "filename" is the name of the unittests executable.

Sounds interesting.
 That way both the
 unittest and the regular executables will be created at the same time -
 which will speed things up, as most of the compilation work can be
 shared between those two builds - and have different file names, so the
 build system - which will make configuring the build system more
 straightforward.

Yeppers. One other thought I had was to define a special flag e.g. --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".
 If we make the unittest a separate executable, there won't be a `main`
 function waiting for command line arguments, so we can use the command
 line arguments for the unittests. We can have flags for only running
 tests of specific packages, or for running the tests under certain
 restrictions etc.

There we go :o). Andrei
Jan 26 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 8:08 PM, Jesse Phillips wrote:
 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:
 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".

I really think it would be better to use --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent clearer.

I'm just saying it should not clash with any application argument. Andrei
Jan 26 2014
next sibling parent reply Alix Pexton <alix.DOT.pexton gmail.DOT.com> writes:
On 27/01/2014 4:15 AM, Andrei Alexandrescu wrote:
 On 1/26/14 8:08 PM, Jesse Phillips wrote:
 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:
 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".

I really think it would be better to use --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent clearer.

I'm just saying it should not clash with any application argument. Andrei

Hows about making it so that unittests are only run if the executable name has a "_ut" suffix, or some other special name/convention? A... --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com
Jan 27 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/27/14 3:10 AM, Alix Pexton wrote:
 On 27/01/2014 4:15 AM, Andrei Alexandrescu wrote:
 On 1/26/14 8:08 PM, Jesse Phillips wrote:
 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:
 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".

I really think it would be better to use --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent clearer.

I'm just saying it should not clash with any application argument. Andrei

Hows about making it so that unittests are only run if the executable name has a "_ut" suffix, or some other special name/convention? A...

Something like that would work nicely with a build system - the executable name has a specific suffix and is promoted to the non-unittest suffix once it has passed unittests. Andrei
Jan 27 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 10:08 PM, H. S. Teoh wrote:
 On Sun, Jan 26, 2014 at 08:15:23PM -0800, Andrei Alexandrescu wrote:
 On 1/26/14 8:08 PM, Jesse Phillips wrote:
 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:
 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".

I really think it would be better to use --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent clearer.

I'm just saying it should not clash with any application argument.

That won't work if the application is a git client looking for an SHA hash argument...

Of course it works unless there's a clash with this particular SHA.
 Let's not add arbitrary things like this to D programs, it doesn't
 address the issue and only introduces needless special cases.

I don't see that a special case. The only idea here is to produce one single debug executable containing unittests and running them on demand (e.g. by the build system). Andrei
Jan 27 2014
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 25 Jan 2014 17:55:30 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 What do you think? Logistically it shouldn't be too hard to arrange  
 things to cater to this approach.

After reading other opinions, this is what I think: 1. unit tests should be built as a separate binary. So when you build foo with -unittest, you get foo_unittest in addition to foo. 2. If you want to run unit tests, run foo_unittest. If you want simply the program, run foo. I don't see a huge need for having unit tests run before the normal program. But maybe we should add a switch to make that work. Building and running it as part of compilation seems like an incorrect function of the compiler. This is best left to an IDE/build script. Also, building unit tests still needs to be opt-in. Some projects can take a long time to build unit tests (dcollections takes about 20x longer to compile unit tests, last time I checked), and a quick compile-test-debug cycle is a key feature of D. -Steve
Jan 26 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 12:43 PM, Steven Schveighoffer wrote:
 On Sat, 25 Jan 2014 17:55:30 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.

After reading other opinions, this is what I think: 1. unit tests should be built as a separate binary. So when you build foo with -unittest, you get foo_unittest in addition to foo. 2. If you want to run unit tests, run foo_unittest. If you want simply the program, run foo.

Makes sense, though we could massage the unittests into the executable.
 I don't see a huge need for having unit tests run before the normal
 program. But maybe we should add a switch to make that work.

Guess we'd need to keep things running as they do for backwards compatibility.
 Building and running it as part of compilation seems like an incorrect
 function of the compiler. This is best left to an IDE/build script.
 Also, building unit tests still needs to be opt-in. Some projects can
 take a long time to build unit tests (dcollections takes about 20x
 longer to compile unit tests, last time I checked), and a quick
 compile-test-debug cycle is a key feature of D.

Yah it should be all nicely controllable by the build system. Andrei
Jan 26 2014
prev sibling next sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei

I don't fully understand this. Are you proposing to disconnect running unittest and evoking main? So unittests can be compiled and run without actually running the program? If this is the case then yes it would be nice to be able to do this but i would also recommend keeping the current behaviour of invoking main after the unittests have run, purely because developers (myself included) like to run the application often in debug/unittest mode during development.
Jan 26 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 1:04 PM, Gary Willoughby wrote:
 I don't fully understand this. Are you proposing to disconnect running
 unittest and evoking main? So unittests can be compiled and run without
 actually running the program?

Right.
 If this is the case then yes it would be nice to be able to do this but
 i would also recommend keeping the current behaviour of invoking main
 after the unittests have run, purely because developers (myself
 included) like to run the application often in debug/unittest mode
 during development.

We'd need to preserve backward compatibility. Andrei
Jan 26 2014
prev sibling next sibling parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.

I can imagine someone who discovered a bug late at night, has a fix and needs to upload the new executable as soon as possible: he quickly comments all failing unit tests to make them pass. The next morning he uncomments them and fixes them with tranquility.
Jan 26 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 5:36 PM, Ary Borenszweig wrote:
 On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.

I can imagine someone who discovered a bug late at night, has a fix and needs to upload the new executable as soon as possible: he quickly comments all failing unit tests to make them pass. The next morning he uncomments them and fixes them with tranquility.

The point being? Andrei
Jan 26 2014
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 1/27/14, 1:04 AM, Andrei Alexandrescu wrote:
 On 1/26/14 5:36 PM, Ary Borenszweig wrote:
 On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.

I can imagine someone who discovered a bug late at night, has a fix and needs to upload the new executable as soon as possible: he quickly comments all failing unit tests to make them pass. The next morning he uncomments them and fixes them with tranquility.

The point being? Andrei

That it's annoying if you can't build an executable because some tests fail. Sometimes you know tests fail but you didn't have time to fix them (but you did fix the code).
Jan 27 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/27/14 3:39 AM, Ary Borenszweig wrote:
 On 1/27/14, 1:04 AM, Andrei Alexandrescu wrote:
 On 1/26/14 5:36 PM, Ary Borenszweig wrote:
 On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.

I can imagine someone who discovered a bug late at night, has a fix and needs to upload the new executable as soon as possible: he quickly comments all failing unit tests to make them pass. The next morning he uncomments them and fixes them with tranquility.

The point being? Andrei

That it's annoying if you can't build an executable because some tests fail. Sometimes you know tests fail but you didn't have time to fix them (but you did fix the code).

Got it, thanks. Andrei
Jan 27 2014
prev sibling next sibling parent =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On 2014-01-25 23:06, Jakob Ovrum wrote:
 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
 In particular, this view of unittests declares our current stance on
 running unittests ("run unittests just before main()") as meaningless.
 Indeed that has bothered me for quite a while - unittests are part of
 the build/acceptance, not part of every run. To wit, this is a growing
 idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

This idiom is probably mostly obsolete now that we have the -main flag. An IDE can build with -unittest -main -run and exclude the source file that normally defines `main`.

Except that would exclude any unit tests in the module that defines main. Of course, if you want to be evil, you could even write unit tests for main: int main() { return 3; } unittest { assert(main() == 3); } -- Simen
Jan 26 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 27 January 2014 at 03:45:00 UTC, Andrei Alexandrescu 
wrote:
 On 1/25/14 7:52 PM, Jesse Phillips wrote:
 I already attempt to get unittests to fail at compile time by 
 using
 static assert. I think running it during compilation would be 
 a great
 change.

That's great but I wasn't thinking of CTFEing unittests as much as just running them right after building. Andrei

I'm aware of that, my point being that I'm already trying to get them to run as frequently as possible.
Jan 26 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu 
wrote:
 Yeppers. One other thought I had was to define a special flag 
 e.g. --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run 
 unittests".

I really think it would be better to use --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent clearer.
Jan 26 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 26, 2014 at 07:53:30PM -0800, Andrei Alexandrescu wrote:
 On 1/26/14 3:25 AM, simendsjo wrote:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:
There's this simple realization that unittests could (should?) be
considered an intrinsic part of the build process.

In particular, this view of unittests declares our current stance on
running unittests ("run unittests just before main()") as
meaningless.

I wouldn't mind having unittests be a part of the compilation process, but I really don't think "running before main" is useless. This lets me compile a version on my desktop and push the binary to a small VPS staging for running runtime tests. Compiling any non-trivial code on 512MB with the DMD frontend is impossible. I can also send a fully working application with unittests to a client as a beta before giving a production version.

Does it help that the client runs them every time? I.e. does the n+1th run of the unittests bring more information than the first?

It's very useful when you're in a code-compile-run development cycle, when you're likely to be making some further changes after every execution. Any regressions are quickly caught by unittests. Obviously, it's useless to run the same unittests twice when nothing has changed, but then, I thought the idea was that you *don't* compile with -unittest when shipping the executable. T -- Once the bikeshed is up for painting, the rainbow won't suffice. -- Andrei Alexandrescu
Jan 26 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 26, 2014 at 08:15:23PM -0800, Andrei Alexandrescu wrote:
 On 1/26/14 8:08 PM, Jesse Phillips wrote:
On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:
Yeppers. One other thought I had was to define a special flag e.g.
--4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".

I really think it would be better to use --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent clearer.

I'm just saying it should not clash with any application argument.

That won't work if the application is a git client looking for an SHA hash argument... Let's not add arbitrary things like this to D programs, it doesn't address the issue and only introduces needless special cases. T -- "Hi." "'Lo."
Jan 26 2014
prev sibling next sibling parent Knud Soerensen <4tuu4k002 sneakemail.com> writes:
I am already doing something very similar.

When I make a module I add this to the top.
#!/usr/bin/rdmd --shebang -unittest --main
(I would like to add -cov=100 but I can't get it to work)
Then I just run the module to compile and run unit tests.

My programs with main is always very small,
typical calling a module function with the arguments.

Doing the unit testing in the end of the compile would
mean that we could ensure that dependencies  was unit testing before
use. That would be nice.

Another thing I find really tedious about D unit testing can you read
about here.

http://forum.dlang.org/post/lc514a$sd7$1 digitalmars.com

Knud



On 2014-01-25 23:55, Andrei Alexandrescu wrote:
 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.
 
 In that view, asserts inside unittests should fail with the same message
 format as regular compilation errors, i.e.
 
 ./modulename.d(103): Unittest failed: user-defined message
 
 Stack traces and other artifacts printed by failing asserts should come
 after, indented. An IDE user should run unittest with the usual "build"
 command and be able to click on the unittest failures just the same as
 build errors.
 
 In particular, this view of unittests declares our current stance on
 running unittests ("run unittests just before main()") as meaningless.
 Indeed that has bothered me for quite a while - unittests are part of
 the build/acceptance, not part of every run. To wit, this is a growing
 idiom in D programs:
 
 version(unittest) void main() {}
 else void main()
 {
     ...
 }
 
 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.
 
 
 Andrei

-- Join me on Skype knudhs Facebook http://www.facebook.com/profile.php?id=1198821880 Linkedin http://www.linkedin.com/pub/0/117/a54 Twitter http://twitter.com/knudsoerensen bitcoin donations: 13ofyUKqFL43uRJHZtNozyMVP4qxKPsAR2
Jan 26 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
I share the opinion others have expressed here that running unit 
tests should be the responsability of the build system.

Atila

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:
 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei

Jan 27 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
 +1. Some of my unittests can only be run at runtime: like 
 testing file
 I/O.  Obviously, other unittests can also be run at 
 compile-time, so
 it's useful to have both.

Tests that do I/O aren't unit tests. They're more likely integration tests. Atila
Jan 27 2014
prev sibling next sibling parent "Idan Arye" <GenericNPC gmail.com> writes:
On Monday, 27 January 2014 at 11:10:04 UTC, Alix Pexton wrote:
 On 27/01/2014 4:15 AM, Andrei Alexandrescu wrote:
 On 1/26/14 8:08 PM, Jesse Phillips wrote:
 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei 
 Alexandrescu wrote:
 Yeppers. One other thought I had was to define a special 
 flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run 
 unittests".

I really think it would be better to use --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent clearer.

I'm just saying it should not clash with any application argument. Andrei

Hows about making it so that unittests are only run if the executable name has a "_ut" suffix, or some other special name/convention? A...

How about exposing the sybmol of the function that runs the unittest, and having a "dunittest" tool for running the tests stored inside a regular executable? (I think it's possible?)
Jan 27 2014
prev sibling next sibling parent =?windows-1252?Q?Simen_Kj=E6r=E5s?= <simen.kjaras gmail.com> writes:
On 2014-01-25 22:55, Andrei Alexandrescu wrote:
[Good stuff]
 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.

As evinced by Bugzilla 5091[0], I always run unit tests separately from the rest of the program, and so fully support this motion. As others have pointed out, there are possible problems with cross-compilation, but all in all, I think treating unit tests as a build step rather than something to do with execution is a Good Thing. [0]: https://d.puremagic.com/issues/show_bug.cgi?id=5091 -- Simen
Jan 27 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 27 January 2014 at 12:18:55 UTC, Idan Arye wrote:
 How about exposing the sybmol of the function that runs the 
 unittest, and having a "dunittest" tool for running the tests 
 stored inside a regular executable? (I think it's possible?)

It is already possible via library solution.
Jan 27 2014