www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Parallel execution of unittests

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Hello,


A coworker mentioned the idea that unittests could be run in parallel 
(using e.g. a thread pool). I've rigged things to run in parallel 
unittests across modules, and that works well. However, this is too 
coarse-grained - it would be great if each unittest could be pooled 
across the thread pool. That's more difficult to implement.

This brings up the issue of naming unittests. It's becoming increasingly 
obvious that anonymous unittests don't quite scale - coworkers are 
increasingly talking about "the unittest at line 2035 is failing" and 
such. With unittests executing in multiple threads and issuing e.g. 
logging output, this is only likely to become more exacerbated. We've 
resisted named unittests but I think there's enough evidence to make the 
change.

Last but not least, virtually nobody I know runs unittests and then 
main. This is quickly becoming an idiom:

version(unittest) void main() {}
else void main()
{
    ...
}

I think it's time to change that. We could do it the 
non-backward-compatible way by redefining -unittest to instruct the 
compiler to not run main. Or we could define another flag such as 
-unittest-only and then deprecate the existing one.

Thoughts? Would anyone want to work on such stuff?


Andrei
Apr 30 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in 
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}
 We've resisted named unittests but I think there's enough
 evidence to make the change.

Yes, the optional name for unittests is an improvement: unittest {} unittest foo {} I am very glad your coworker find such usability problems :-)
 We could do it the non-backward-compatible way by
 redefining -unittest to instruct the compiler to not run main.

Good. I'd also like some built-in way (or partially built-in) to use a module only as "main module" (to run its demos) or as module to be imported. This problem is solved in Python with the "if __name__ == "__main__":" idiom. Bye, bearophile
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei
Apr 30 2014
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message 
news:ljr6ld$1mft$2 digitalmars.com...

 This doesn't follow. All unittests should be executable concurrently. -- 
 Andrei

That's like saying all makefiles should work with -j
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 11:13 AM, Daniel Murphy wrote:
 "Andrei Alexandrescu"  wrote in message
 news:ljr6ld$1mft$2 digitalmars.com...

 This doesn't follow. All unittests should be executable concurrently.
 -- Andrei

That's like saying all makefiles should work with -j

They should. -- Andrei
Apr 30 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 10:50 AM, Jonathan M Davis via Digitalmars-d wrote:
 There
 is nothing whatsoever in the language which guarantees that running
 them in parallel will work or even makes sense.

Default thread-local globals? -- Andrei
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:25 PM, Jonathan M Davis via Digitalmars-d wrote:
 Sure, that helps, but it's trivial to write a unittest block which
 depends on a previous unittest block, and as soon as a unittest block
 uses an external resource such as a socket or file, then even if a
 unittest block doesn't directly depend on the end state of a
 previous unittest block, it still depends on external state which could
 be affected by other unittest blocks. So, ultimately, the language
 really doesn't ensure that running a unittest block can be
 parallelized. If it's pure as bearophile suggested, then it can be
 done, but as long as a unittest block is impure, then it can rely on
 global state - even inadvertently - (be it state directly in the program
 or state outside the program) and therefore not work when pararellized.
 So, I suppose that you could parallelize unittest blocks if they were
 marked as pure (though I'm not sure if that's currently a legal thing
 to do), but impure unittest blocks aren't guaranteed to be
 parallelizable.

Agreed. I think we should look into parallelizing all unittests. -- Andrei
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 10:01 PM, Jonathan M Davis via Digitalmars-d wrote:
 I'm all for parallelizing all unittest blocks that are pure, as doing
 so would be safe, but I think that we're making a big mistake if we try
 and insist that all unittest blocks be able to be run in parallel. Any
 that aren't pure are not guaranteed to be parallelizable, and any which
 access system resources or other global, mutable state stand a good
 chance of breaking.

There are a number of assumptions here: (a) most unittests that can be effectively parallelized can be actually inferred (or declared) as pure; (b) most unittests that cannot be inferred as pure are likely to break; (c) it's a big deal if unittests break. I question all of these assumptions. In particular I consider unittests that depend on one another an effective antipattern that needs to be eradicated. Andrei
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 11:31 PM, Jonathan M Davis via Digitalmars-d wrote:
 On Wed, 30 Apr 2014 22:32:33 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:
 There are a number of assumptions here: (a) most unittests that can
 be effectively parallelized can be actually inferred (or declared) as
 pure; (b) most unittests that cannot be inferred as pure are likely
 to break; (c) it's a big deal if unittests break. I question all of
 these assumptions. In particular I consider unittests that depend on
 one another an effective antipattern that needs to be eradicated.

Even if they don't depend on each other, they can depend on the system.

Understood, no need to repeat, thanks.
 std.file's unit tests will break if we parallelize them, because it
 operates on files and directories, and many of those tests operate on
 the same temp directories.

Yah, I remember even times when make unittest -j broke unittests because I've used the file name "deleteme" in multiple places. We need to fix those.
 That can be fixed by changing the tests, but
 it will break the tests.

I'm not too worried about breaking tests. I have in mind that we'll display a banner at the beginning of unittesting explaining that tests are ran in parallel and to force serial execution they'd need to set this thing or that. In a way I don't see it as "breakage" in the traditional tests. Unittests are in a way supposed to break :o).
 Other tests _can't_ be fixed if we force them
 to run in parallel. For instance, some of std.datetime's unit tests set
 the local time zone of the system in order to test that LocalTime works
 correctly.

Sure. We could specify that tests are to be run serially within one specific module, or to use classic interlocking in the unittest code. I see it as a problem relatively easy to address.
 We can guarantee that pure functions can safely be run in parallel. We
 _cannot_ guarantee that impure functions can safely be run in parallel.
 I'm sure that many impure unittest functions could be safely run in
 parallel, but it would require that the programmer verify that if we
 don't want undefined behavior - just like programmers have to verify
 that  system code is actually  safe. Simply running all unittest blocks
 in parallel is akin to considering  system code  safe in a particular
 piece of code simply because by convention that code should be  safe.

I don't think undefined behavior is at stake here, and I find the simile invalid. Thread isolation is a done deal in D and we may as well take advantage of it. Worse that could happen is that a unittest sets a global and surprisingly the next one doesn't "see" it. At any rate I think it's pointless to insist on limiting parallel running to pure - let me just say I understood the point (thanks) so there is no need to restate it, and that I think it doesn't take us a good place. Andrei
Apr 30 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. -- Andrei

I recommend running the tests in random order as well. -- /Jacob Carlborg
May 01 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 4:05 AM, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

Great idea! -- Andrei
May 01 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 4:44 AM, w0rp wrote:
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

I do random testing all the time, and I print the seed of the prng upon startup. When something fails randomly, I take the seed and seed the prng with it to reproduce. -- Andrei
May 01 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 01/05/2014 13:44, w0rp a écrit :
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

May 01 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 01/05/2014 16:01, Atila Neves a écrit :
 On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that. Atila

May 01 2014
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-05-01 19:12, Steven Schveighoffer wrote:

 But not a great way to debug it.

 If your test failure depends on ordering, then the next run will be
 random too.

 Proposal runtime parameter for pre-main consumption:

 ./myprog --rndunit[=seed]

 To run unit tests randomly. Prints out as first order of business the
 seed value before starting. That way, you can repeat the exact same
 ordering for debugging.

That's exactly what RSpec does. I think it works great. -- /Jacob Carlborg
May 01 2014
prev sibling parent Bruno Medeiros <bruno.do.medeiros+dng gmail.com> writes:
On 01/05/2014 18:12, Steven Schveighoffer wrote:
 On Thu, 01 May 2014 10:01:19 -0400, Atila Neves <atila.neves gmail.com>
 wrote:

 On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that.

But not a great way to debug it. If your test failure depends on ordering, then the next run will be random too.

I agree with Steven here. Actually, even if the failure does *not* depend on ordering, it can still be useful to run the tests in order when debugging: If there is a bug in a low level component, that will likely trigger a failure in the tests for that low level component, but also the tests for higher-level components (the components that use the low level component). As such, when debugging, you would want to run the low-level test first since it will likely be easier to debug the issue there, than with the higher-level test. Sure, one could say that the solution to this should be mocking the low-level component in the high-level test, but mocking is not always desirable or practical. I can provide some concrete examples. -- Bruno Medeiros https://twitter.com/brunodomedeiros
May 06 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-04-30 23:25, Jonathan M Davis via Digitalmars-d wrote:

 Sure, that helps, but it's trivial to write a unittest block which
 depends on a previous unittest block

There for the tests should be run in random order. -- /Jacob Carlborg
May 01 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 10:58 AM, Atila Neves wrote:
 We could always make running in threads opt-in.

Yah, great idea. -- Andrei
Apr 30 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:09 PM, Russel Winder via Digitalmars-d wrote:
 And even then all tests can
 and should be parallelized. If they cannot be then there is an
 inappropriate dependency.

Agreed. -- Andrei
Apr 30 2014
prev sibling next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 30/04/2014 17:59, Andrei Alexandrescu a écrit :
 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

initialized before them. File system operations are generally a critical point, if many unittest are based on same auto-generated file data it's a good idea to run this generation once before all tests (they eventually do a file copy that is fast with copy-on-write file system or those data can be used as read only by all tests). So for those kind of situation some functions have must be able to run before unittest, and I think that the case of static this() function of modules?
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 6:20 PM, Xavier Bigand wrote:
 Le 30/04/2014 17:59, Andrei Alexandrescu a écrit :
 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

initialized before them. File system operations are generally a critical point, if many unittest are based on same auto-generated file data it's a good idea to run this generation once before all tests (they eventually do a file copy that is fast with copy-on-write file system or those data can be used as read only by all tests). So for those kind of situation some functions have must be able to run before unittest, and I think that the case of static this() function of modules?

Yah version(unittest) static shared this() { ... } covers that. -- Andrei
Apr 30 2014
parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 01/05/2014 03:54, Andrei Alexandrescu a écrit :
 On 4/30/14, 6:20 PM, Xavier Bigand wrote:
 Le 30/04/2014 17:59, Andrei Alexandrescu a écrit :
 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

initialized before them. File system operations are generally a critical point, if many unittest are based on same auto-generated file data it's a good idea to run this generation once before all tests (they eventually do a file copy that is fast with copy-on-write file system or those data can be used as read only by all tests). So for those kind of situation some functions have must be able to run before unittest, and I think that the case of static this() function of modules?

Yah version(unittest) static shared this() { ... } covers that. -- Andrei

It stay the question of name, I don't really know if it have to be in the language or in phobos like other tests features (test-logger, benchmark,...).
Apr 30 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 30/04/2014 19:50, Jonathan M Davis via Digitalmars-d a crit :
 On Wed, 30 Apr 2014 08:59:42 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

In general, I agree. In reality, there are times when having state across unit tests makes sense - especially when there's expensive setup required for the tests. While it's not something that I generally like to do, I know that we have instances of that where I work. Also, if the unit tests have to deal with shared resources, they may very well be theoretically independent but would run afoul of each other if run at the same time - a prime example of this would be std.file, which has to operate on the file system. I fully expect that if std.file's unit tests were run in parallel, they would break. Unit tests involving sockets would be another type of test which would be at high risk of breaking, depending on what sockets they need. Honestly, the idea of running unit tests in parallel makes me very nervous. In general, across modules, I'd expect it to work, but there will be occasional cases where it will break. Across the unittest blocks in a single module, I'd be _very_ worried about breakage. There is nothing whatsoever in the language which guarantees that running them in parallel will work or even makes sense. All that protects us is the convention that unit tests are usually independent of each other, and in my experience, it's common enough that they're not independent that I think that blindly enabling parallelization of unit tests across a single module is definitely a bad idea. - Jonathan M Davis

pure unittest name {} seems a good idea, and it's more intuitive to have the same behaviour of other functions with a closer signature.
Apr 30 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 30/04/2014 19:58, Atila Neves a écrit :
 On Wednesday, 30 April 2014 at 17:50:34 UTC, Jonathan M Davis via
 Digitalmars-d wrote:
 On Wed, 30 Apr 2014 08:59:42 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

In general, I agree. In reality, there are times when having state across unit tests makes sense - especially when there's expensive setup required for the tests. While it's not something that I generally like to do, I know that we have instances of that where I work. Also, if the unit tests have to deal with shared resources, they may very well be theoretically independent but would run afoul of each other if run at the same time - a prime example of this would be std.file, which has to operate on the file system. I fully expect that if std.file's unit tests were run in parallel, they would break. Unit tests involving sockets would be another type of test which would be at high risk of breaking, depending on what sockets they need. Honestly, the idea of running unit tests in parallel makes me very nervous. In general, across modules, I'd expect it to work, but there will be occasional cases where it will break. Across the unittest blocks in a single module, I'd be _very_ worried about breakage. There is nothing whatsoever in the language which guarantees that running them in parallel will work or even makes sense. All that protects us is the convention that unit tests are usually independent of each other, and in my experience, it's common enough that they're not independent that I think that blindly enabling parallelization of unit tests across a single module is definitely a bad idea. - Jonathan M Davis

You're right; blindly enabling parallelisation after the fact is likely to cause problems. Unit tests though, by definition (and I'm aware there are more than one) have to be independent. Have to not touch the filesystem, or the network. Only CPU and RAM. In my case, and since I had the luxury of implementing a framework first and only writing tests after it was done, running them in parallel was an extra check that they are in fact independent.

just can't have a good code coverage on a lot libraries with a such restriction. I had work on a Source Control Management software, and all tests have to deal with a DB which requires file system and network operations. IMO it's pretty much like impossible to miss testing of functions relations, simple integration tests are often needed to ensure that the application is working correctly. If D integrate features to support automatized testing maybe it must not be to restrictive mainly if everybody will expect more features commonly used (like named tests, formated result output,...). Some of those common features have to be added to phobos instead of the language.
 Now, it does happen that you're testing code that isn't thread-safe
 itself, and yes, in that case you have to run them in a single thread.
 That's why I added the  SingleThreaded UDA to my library to enable that.
 As soon as I tried calling legacy C code...

 We could always make running in threads opt-in.

 Atila

Apr 30 2014
prev sibling next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 30/04/2014 21:23, Dicebot a écrit :
 On Wednesday, 30 April 2014 at 18:19:34 UTC, Jonathan M Davis via
 Digitalmars-d wrote:
 On Wed, 30 Apr 2014 17:58:34 +0000
 Atila Neves via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 Unit tests though, by definition (and I'm aware there are more than
 one) have to be independent. Have to not touch the filesystem, or the
 network. Only CPU and RAM.

I disagree with this. A unit test is a test that tests a single piece of functionality - generally a function - and there are functions which have to access the file system or network.

They _use_ access to file system or network, but it is _not_ their functionality. Unit testing is all about verifying small perfectly separated pieces of functionality which don't depend on correctness / stability of any other functions / programs. Doing I/O goes against it pretty much by definition and is unfortunately one of most common testing antipatterns.

Splitting all features at an absolute atomic level can be achieve for open-source libraries, but it's pretty much impossible for an industrial software. Why being so restrictive when it's possible to support both vision by extending a little the language by something already logical?
Apr 30 2014
parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 01/05/2014 09:23, Dicebot a écrit :
 On Thursday, 1 May 2014 at 01:45:21 UTC, Xavier Bigand wrote:
 Splitting all features at an absolute atomic level can be achieve for
 open-source libraries, but it's pretty much impossible for an
 industrial software. Why being so restrictive when it's possible to
 support both vision by extending a little the language by something
 already logical?

You are pretty much saying here "writing good code is possible for open-source libraries but not for industrial software".

I am working for a very small company and our dead lines clearly doesn't help us with that, and because I am in the video game industry it's not really critical to have small bugs. Not every body have the capacity or resources (essentially time) to design his code in the pure conformance of unittests definition, and IMO isn't not an excuse to avoid tests completely. If a language/standard library can help democratization of tests it's a good thing, so maybe writing tests have to stay relatively simple and straightforward. My point is just when you are doing things only for you it's often simpler to them like they must be.
May 01 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 30/04/2014 22:09, Russel Winder via Digitalmars-d a écrit :
 On Wed, 2014-04-30 at 11:19 -0700, Jonathan M Davis via Digitalmars-d
 wrote:
 […]
 I disagree with this. A unit test is a test that tests a single piece
 of functionality - generally a function - and there are functions which
 have to access the file system or network. And those tests are done in

These are integration/system tests not unit tests. For unit tests network activity should be mocked out.

to date when changing the code but not the running application cause before the commit you'll run only your unittests. IMO every tests can be automatize and run in a few time have to be run before each commit even if some are integration tests.
 unittest blocks just like any other unit test. I would very much
 consider std.file's tests to be unit tests. But even if you don't
 want to call them unit tests, because they access the file system, the
 reality of the matter is that tests like them are going to be run in
 unittest blocks, and we have to take that into account when we decide
 how we want unittest blocks to be run (e.g. whether they're
 parallelizable or not).

In which case D is wrong to allow them in the unittest blocks and should introduce a new way of handling these tests. And even then all tests can and should be parallelized. If they cannot be then there is an inappropriate dependency.

Apr 30 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal projects removing
 all accidental I/O tests for the very reason that /tmp was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 8:04 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal projects removing
 all accidental I/O tests for the very reason that /tmp was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics?

I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei
May 01 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 9:07 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 8:04 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal projects
 removing
 all accidental I/O tests for the very reason that /tmp was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics?

I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei

It got full because of tests (surprise!). Your actions?

Fix the machine and reduce the output created by the unittests. It's a simple engineering problem. -- Andrei
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 8:11 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 16:08:23 UTC, Andrei Alexandrescu wrote:
 It got full because of tests (surprise!). Your actions?

Fix the machine and reduce the output created by the unittests. It's a simple engineering problem. -- Andrei

You can't. You have not control over that machine you don't even exactly know that test has failed because of full /tmp/ - all you got is a bug report that can't be reproduced on your machine. It is not that simple already and it can get damn complicated once you get to something like network I/O

I know, incidentally the hhvm team has had the same problem two weeks ago. They fixed it (wthout removing file I/O from unittests). It's fixable. That's it. This segment started with your claim that unittests should do no file I/O because they may fail with a full /tmp/. I disagree with that, and with framing the full /tmp/ problem as a problem with the unittests doing file I/O. Andrei
May 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 8:55 AM, Dicebot wrote:
 It was just a most simple example. "Unittests should do no I/O because
 any sort of I/O can fail because of reasons you don't control from the
 test suite" is an appropriate generalization of my statement.

 Full /tmp is not a problem, there is nothing broken about system with
 full /tmp. Problem is test reporting that is unable to connect failure
 with /tmp being full unless you do environment verification.

Different strokes for different folks. -- Andrei
May 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 10:08 AM, Dicebot wrote:
 On Monday, 5 May 2014 at 16:33:42 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 8:55 AM, Dicebot wrote:
 It was just a most simple example. "Unittests should do no I/O because
 any sort of I/O can fail because of reasons you don't control from the
 test suite" is an appropriate generalization of my statement.

 Full /tmp is not a problem, there is nothing broken about system with
 full /tmp. Problem is test reporting that is unable to connect failure
 with /tmp being full unless you do environment verification.

Different strokes for different folks. -- Andrei

There is nothing subjective about it.

Of course there is. -- Andrei
May 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 11:25 AM, Dicebot wrote:
 On Monday, 5 May 2014 at 18:24:43 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 10:08 AM, Dicebot wrote:
 On Monday, 5 May 2014 at 16:33:42 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 8:55 AM, Dicebot wrote:
 It was just a most simple example. "Unittests should do no I/O because
 any sort of I/O can fail because of reasons you don't control from the
 test suite" is an appropriate generalization of my statement.

 Full /tmp is not a problem, there is nothing broken about system with
 full /tmp. Problem is test reporting that is unable to connect failure
 with /tmp being full unless you do environment verification.

Different strokes for different folks. -- Andrei

There is nothing subjective about it.

Of course there is. -- Andrei

You are not helping your point to look reasonable.

My understanding here is you're trying to make dogma out of engineering choices that may vary widely across projects and organizations. No thanks. Andrei
May 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 11:47 AM, Dicebot wrote:
 On Monday, 5 May 2014 at 18:29:40 UTC, Andrei Alexandrescu wrote:
 My understanding here is you're trying to make dogma out of
 engineering choices that may vary widely across projects and
 organizations. No thanks.

 Andrei

I am asking to either suggest an alternative solution or to clarify why you don't consider it is an important problem.

"Clean /tmp/ judiciously."
 Dogmatic approach that
 solves the issue is still better than ignoring it completely.

The problem with your stance, i.e.:
 "Unittests should do no I/O because any sort of I/O can fail because
 of reasons you don't control from the test suite" is an appropriate
 generalization of my statement.

is that it immediately generalizes into the unreasonable: "Unittests should do no $X because any sort of $X can fail because of reasons you don't control from the test suite". So that gets into machines not having any memory available, with full disks etc. Just make sure test machines are prepared for running unittests to the extent unittests are expecting them to. We're wasting time trying to frame this as a problem purely related to unittests alone.
 Right now I am afraid you will push for quick changes that will reduce
 elegant simplicity of D unittest system without providing a sound
 replacement that will actually fit into more ambitious use cases (as
 whole "parallel" thing implies).

If I had my way I'd make parallel the default and single-threaded opt-in, thus penalizing unittests that had issues to start with. But I understand the merits of not breaking backwards compatibility so probably we should start with opt-in parallel unittesting. Andrei
May 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/6/14, 10:43 AM, Dicebot wrote:
 Disk I/O failure throws Exception which can be easily consumed somewhere
 inside tested control flow resulting in absolutely mysterious test failures.

If you're pointing out full /tmp/ should be nicely diagnosed by the unittest, I agree. -- Andrei
May 06 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/6/14, 11:27 AM, Dicebot wrote:
 On Tuesday, 6 May 2014 at 18:13:01 UTC, Andrei Alexandrescu wrote:
 On 5/6/14, 10:43 AM, Dicebot wrote:
 Disk I/O failure throws Exception which can be easily consumed somewhere
 inside tested control flow resulting in absolutely mysterious test
 failures.

If you're pointing out full /tmp/ should be nicely diagnosed by the unittest, I agree. -- Andrei

Good we have a common base ground :) It inevitably arises next question though : how unittest can diagnose it?

Fail with diagnostic. -- Andrei
May 06 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 10:09 AM, Steven Schveighoffer wrote:
 On Thu, 01 May 2014 12:07:19 -0400, Dicebot <public dicebot.lv> wrote:

 On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 8:04 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal projects
 removing
 all accidental I/O tests for the very reason that /tmp was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics?

I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei

It got full because of tests (surprise!). Your actions?

It would be nice to have a uniform mechanism to get a unique system-dependent file location for each specific unit test. The file should automatically delete itself at the end of the test.

Looks like /tmp (%TEMP% or C:\TEMP in Windows) in conjunction with the likes of mkstemp is what you're looking for :o). Andrei
May 01 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 10:32 AM, Brad Anderson wrote:
 It hasn't been C:\TEMP for almost 13 years

About the time when I switched :o). -- Andrei
May 01 2014
prev sibling next sibling parent Bruno Medeiros <bruno.do.medeiros+dng gmail.com> writes:
On 30/04/2014 20:23, Dicebot wrote:
 On Wednesday, 30 April 2014 at 18:19:34 UTC, Jonathan M Davis via
 Digitalmars-d wrote:
 On Wed, 30 Apr 2014 17:58:34 +0000
 Atila Neves via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 Unit tests though, by definition (and I'm aware there are more than
 one) have to be independent. Have to not touch the filesystem, or the
 network. Only CPU and RAM.

I disagree with this. A unit test is a test that tests a single piece of functionality - generally a function - and there are functions which have to access the file system or network.

They _use_ access to file system or network, but it is _not_ their functionality. Unit testing is all about verifying small perfectly separated pieces of functionality which don't depend on correctness / stability of any other functions / programs. Doing I/O goes against it pretty much by definition and is unfortunately one of most common testing antipatterns.

It is common, but it is not necessarily an anti-pattern. Rather it likely is just an Integration test instead of a Unit test. See: http://forum.dlang.org/post/lkb0jm$vp8$1 digitalmars.com -- Bruno Medeiros https://twitter.com/brunodomedeiros
May 06 2014
prev sibling parent Bruno Medeiros <bruno.do.medeiros+dng gmail.com> writes:
On 01/05/2014 08:18, Dicebot wrote:
 On Wednesday, 30 April 2014 at 21:49:06 UTC, Jonathan M Davis via
 Digitalmars-d wrote:
 On Wed, 30 Apr 2014 21:09:14 +0100
 Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wed, 2014-04-30 at 11:19 -0700, Jonathan M Davis via Digitalmars-d
 wrote:
 unittest blocks just like any other unit test. I would very > much
 consider std.file's tests to be unit tests. But even if you > don't
 want to call them unit tests, because they access the file > system,
 the reality of the matter is that tests like them are going > to be
 run in unittest blocks, and we have to take that into > account when
 we decide how we want unittest blocks to be run (e.g. whether
 they're parallelizable or not).

In which case D is wrong to allow them in the unittest blocks and should introduce a new way of handling these tests. And even then all tests can and should be parallelized. If they cannot be then there is an inappropriate dependency.

Why? Because Andrei suddenly proposed that we parallelize unittest blocks? If I want to test a function, I'm going to put a unittest block after it to test it. If that means accessing I/O, then it means accessing I/O. If that means messing with mutable, global variables, then that means messing with mutable, global variables. Why should I have to put the tests elsewhere or make is that they don't run whenthe -unttest flag is used just because they don't fall under your definition of "unit" test?

You do this because unit tests must be fast. You do this because unit tests must be naively parallel. You do this because unit tests verify basic application / library sanity and expected to be quickly run after every build in deterministic way (contrary to full test suite which can take hours).

See http://forum.dlang.org/post/lkb0jm$vp8$1 digitalmars.com. (basically, do we want to support only Unit tests, or Integration tests also?) -- Bruno Medeiros https://twitter.com/brunodomedeiros
May 06 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 8:54 AM, bearophile wrote:
 I'd also like some built-in way (or partially built-in) to use a module only as
 "main module" (to run its demos) or as module to be imported. This problem is
 solved in Python with the "if __name__ == "__main__":" idiom.

dmd foo.d -unittest -main
Apr 30 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 12:32 AM, bearophile wrote:
 This is just the basic idea, and perhaps people suggested something better than
 this.

You've already got it working with version, that's what version is for. Why add yet another way to do it?
May 01 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/2/2014 4:02 AM, bearophile wrote:
 Walter Bright:

 You've already got it working with version, that's what version is for. Why
 add yet another way to do it?

Because I'd like something better. It's an idiom that I have used many times (around 15-20 times). I'd like the compiler (or build tool) to avoid me to specify two times what the main module is. Also, the current way to do it, in those modules I have to specify the module name two times (once at the top and once at the bottom, unless I use some compile-time syntheses of the version identifier from the current module name).

D has so many language features, we need a higher bar for adding new ones, especially ones that can be done straightforwardly with existing features.
May 04 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 11:53 AM, monarch_dodra wrote:
 On Wednesday, 30 April 2014 at 15:54:42 UTC, bearophile wrote:
 We've resisted named unittests but I think there's enough
 evidence to make the change.

Yes, the optional name for unittests is an improvement: unittest {} unittest foo {} I am very glad your coworker find such usability problems :-)

If we do "name" the unittests, then can we name them with strings? No need to polute namespace with ugly symbols. Also: //---- unittest "Sort: Non-Lvalue RA range" { ... } //---- vs //---- unittest SortNonLvalueRARange { ... } //----

I'd argue for regular identifiers instead of strings - they can be seen in stack traces, accessed with __FUNCTION__ etc. -- Andrei
Apr 30 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-04-30 21:12, bearophile wrote:

 Are UDAs enough?

  uname("foo") unittest {}

 What I'd like is to tie one or more unittests to other entities, like
 all the unittests of a specific function.

Something similar is done in RSpec. A BDD framework in Ruby. In D it might look like this: describe("foo") { it("does something useful") unittest { } it("also does some other stuff") unittest { } } -- /Jacob Carlborg
May 01 2014
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 30 Apr 2014 08:43:31 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 However, this is too 
 coarse-grained - it would be great if each unittest could be pooled 
 across the thread pool. That's more difficult to implement.

I filed a pull request which allowed running unit tests individually (and in different threads*) two years ago, but didn't pursue this further: https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308 To summarize: It provides a function pointer for every unit test to druntime or user code. This is actually easy to do. Naming tests requires changes in the parser, but I guess that shouldn't be difficult either. * Some time ago there was a discussion whether unit tests can rely on other tests being executed first / execution order. AFAIK some phobos tests require this. That of course won't work if you run the tests in different threads.
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 8:54 AM, Johannes Pfau wrote:
 Am Wed, 30 Apr 2014 08:43:31 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 However, this is too
 coarse-grained - it would be great if each unittest could be pooled
 across the thread pool. That's more difficult to implement.

I filed a pull request which allowed running unit tests individually (and in different threads*) two years ago, but didn't pursue this further: https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308 To summarize: It provides a function pointer for every unit test to druntime or user code. This is actually easy to do. Naming tests requires changes in the parser, but I guess that shouldn't be difficult either.

That's fantastic, would you be willing to reconsider that work?
 * Some time ago there was a discussion whether unit tests can rely on
    other tests being executed first / execution order. AFAIK some phobos
    tests require this. That of course won't work if you run the tests in
    different threads.

I think indeed a small number of unittests rely on order of execution. Those will be still runnable with a fork factor of 1. We'd need a way to specify that - either a flag or: static shared this() { Runtime.unittestThreads = 1; } Andrei
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 9:19 AM, Byron wrote:
 On Wed, 30 Apr 2014 09:02:54 -0700, Andrei Alexandrescu wrote:

 I think indeed a small number of unittests rely on order of execution.
 Those will be still runnable with a fork factor of 1. We'd need a way to
 specify that - either a flag or:

 static shared this() { Runtime.unittestThreads = 1; }


 Andrei

Named tested seems like a no brainier to me. Maybe nested unittests? unittest OrderTests { // setup for all child tests? unittest a { } unittest b { } }

I wouldn't want to get too excited about stuff without there being a need for it. We risk overcomplicating things (i.e what happens inside loops etc).
 I also wonder if its just better to extend/expose the unittest API for
 more advanced things like order of execution, test reporting, and parallel
 execution. And we can just support an external unittesting library to do
 all the advanced testing options.

That would be pretty rad. Andrei
Apr 30 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 30/04/2014 18:19, Byron a écrit :
 On Wed, 30 Apr 2014 09:02:54 -0700, Andrei Alexandrescu wrote:

 I think indeed a small number of unittests rely on order of execution.
 Those will be still runnable with a fork factor of 1. We'd need a way to
 specify that - either a flag or:

 static shared this() { Runtime.unittestThreads = 1; }


 Andrei

Named tested seems like a no brainier to me. Maybe nested unittests? unittest OrderTests { // setup for all child tests? unittest a { } unittest b { } } I also wonder if its just better to extend/expose the unittest API for more advanced things like order of execution, test reporting, and parallel execution. And we can just support an external unittesting library to do all the advanced testing options.

unittest in a tree and using the module's names for the hierarchy.
Apr 30 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 4:35 AM, Johannes Pfau wrote:
 Am Wed, 30 Apr 2014 09:02:54 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 To summarize: It provides a function pointer for every  unit test to
 druntime or user code. This is actually easy to do. Naming tests
 requires changes in the parser, but I guess that shouldn't be
 difficult either.

That's fantastic, would you be willing to reconsider that work?

Are you still interested in this?

Yes.
 I guess we could build a std.test phobos module to completely replace
 the current unittest implementation, see:
 http://forum.dlang.org/post/ljtbch$lg6$1 digitalmars.com

 This seems to be a better solution, especially considering extension
 possibilities.

I think anything that needs more than the user writing unittests and adding a line somewhere would be suboptimal. Basically we need all we have now, just run in parallel. Andrei
May 01 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 10:41 AM, Jason Spencer wrote:
 On Wednesday, 30 April 2014 at 16:19:48 UTC, Byron wrote:
 On Wed, 30 Apr 2014 09:02:54 -0700, Andrei Alexandrescu wrote:

 I think indeed a small number of unittests rely on order of execution.

Maybe nested unittests? unittest OrderTests { // setup for all child tests? unittest a { } unittest b { } }

I like my unit tests to be next to the element under test, and it seems like this nesting would impose some limits on that. Another idea might be to use the level of the unit as an indicator of order dependencies. If UTs for B call/depend on A, then we would assign A to level 0, run it's UTs first, and assign B to level 1. All 0's run before all 1's. Could we use a template arg on the UT to indicate level? unittest!(0) UtA { // test A} unittest!{1} UtB { // test B} Or maybe some fancier compiler dependency analysis?

Well how complicated can we make it all? -- Andrei
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 2:28 PM, Jason Spencer wrote:
 On Thursday, 1 May 2014 at 17:57:05 UTC, Andrei Alexandrescu wrote:
 Well how complicated can we make it all? -- Andrei

As simple as possible, but no simpler :) I've seen you favor this or that feature because it would make unit testing easier and more accessible, and eschew features that would cause folks to not bother. In truth, we could leave it how it is. But I surmise you started this thread to improve the feature and encourage more use of unit test. So we're looking for the sweet spot. I don't think it's important to support the sharing of state between unit tests. But I do see value in being able to influence the order of test execution, largely for debugging reasons. It's important for a module's tests to be able to depend on other modules--otherwise, unittest is not very enticing. If it does and there's a failure, it's hugely helpful to know the failure is caused by the unit-under-test, and not the dependency(s). The common way to do that is to run the tests in reverse order of dependency--i.e. levelize the design and test from the bottom up. See "Large Scale C++ SW Design", Lakos, Chp. 3-4. I imagine there are other niche reasons for order, but for me, this is the driving reason. So it seems a nice middle ground. If order is important, it might be a workable approach to run unittest in the reverse module dependency order by default. A careful programmer could arrange those classes/functions in modules to take advantage of that order if it were important. Seems like we'd have the dependency information--building and traversing a tree shouldn't be that tough.... To preserve it, you'd only be able to parallelize the UTs within a module at a time (unless there's a different flag or something.) But it seems the key question is whether order can EVER be important for any reason. I for one would be willing to give up parallelization to get levelized tests. What are you seeing on your project? How do you allow tests to have dependencies and avoid order issues? Why is parallelization more important than that?

I'll be blunt. What you say is technically sound (which is probably why you believe it is notable) but seems to me an unnecessarily complex engineering contraption that in all likelihood has more misuses than good uses. I fully understand you may think I'm a complete chowderhead for saying this; in the past I've been in your place and others have been in mine, and it took me years to appreciate both positions. -- Andrei
May 01 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 8:04 PM, Jason Spencer wrote:
 On Thursday, 1 May 2014 at 21:40:38 UTC, Andrei Alexandrescu wrote:
 I'll be blunt. What you say is technically sound (which is probably
 why you believe it is notable)...

Well, I suppose that's not the MOST insulting brush-off I could hope for, but it falls short of encouraging me to contribute ideas for the improvement of the language.

Sorry, and that's great. Thanks! -- Andrei
May 02 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 8:04 PM, Jason Spencer wrote:
 On Thursday, 1 May 2014 at 21:40:38 UTC, Andrei Alexandrescu wrote:
 I'll be blunt. What you say is technically sound (which is probably
 why you believe it is notable)...

Well, I suppose that's not the MOST insulting brush-off I could hope for, but it falls short of encouraging me to contribute ideas for the improvement of the language.

I need to make an amend to this because indeed it's more than 2 std deviations away from niceness: I have a long history of ideas with a poor complexity/usefulness ratio, and I now wish I'd received such a jolt. -- Andrei
May 02 2014
prev sibling next sibling parent Byron <byron.heads gmail.com> writes:
On Wed, 30 Apr 2014 09:02:54 -0700, Andrei Alexandrescu wrote:

 
 I think indeed a small number of unittests rely on order of execution.
 Those will be still runnable with a fork factor of 1. We'd need a way to
 specify that - either a flag or:
 
 static shared this() { Runtime.unittestThreads = 1; }
 
 
 Andrei

Named tested seems like a no brainier to me. Maybe nested unittests? unittest OrderTests { // setup for all child tests? unittest a { } unittest b { } } I also wonder if its just better to extend/expose the unittest API for more advanced things like order of execution, test reporting, and parallel execution. And we can just support an external unittesting library to do all the advanced testing options.
Apr 30 2014
prev sibling next sibling parent "Jason Spencer" <j8spencer gmail.com> writes:
On Wednesday, 30 April 2014 at 16:19:48 UTC, Byron wrote:
 On Wed, 30 Apr 2014 09:02:54 -0700, Andrei Alexandrescu wrote:

 
 I think indeed a small number of unittests rely on order of 
 execution.

Maybe nested unittests? unittest OrderTests { // setup for all child tests? unittest a { } unittest b { } }

I like my unit tests to be next to the element under test, and it seems like this nesting would impose some limits on that. Another idea might be to use the level of the unit as an indicator of order dependencies. If UTs for B call/depend on A, then we would assign A to level 0, run it's UTs first, and assign B to level 1. All 0's run before all 1's. Could we use a template arg on the UT to indicate level? unittest!(0) UtA { // test A} unittest!{1} UtB { // test B} Or maybe some fancier compiler dependency analysis?
May 01 2014
prev sibling next sibling parent "Jason Spencer" <j8spencer gmail.com> writes:
On Thursday, 1 May 2014 at 17:57:05 UTC, Andrei Alexandrescu 
wrote:
 Well how complicated can we make it all? -- Andrei

As simple as possible, but no simpler :) I've seen you favor this or that feature because it would make unit testing easier and more accessible, and eschew features that would cause folks to not bother. In truth, we could leave it how it is. But I surmise you started this thread to improve the feature and encourage more use of unit test. So we're looking for the sweet spot. I don't think it's important to support the sharing of state between unit tests. But I do see value in being able to influence the order of test execution, largely for debugging reasons. It's important for a module's tests to be able to depend on other modules--otherwise, unittest is not very enticing. If it does and there's a failure, it's hugely helpful to know the failure is caused by the unit-under-test, and not the dependency(s). The common way to do that is to run the tests in reverse order of dependency--i.e. levelize the design and test from the bottom up. See "Large Scale C++ SW Design", Lakos, Chp. 3-4. I imagine there are other niche reasons for order, but for me, this is the driving reason. So it seems a nice middle ground. If order is important, it might be a workable approach to run unittest in the reverse module dependency order by default. A careful programmer could arrange those classes/functions in modules to take advantage of that order if it were important. Seems like we'd have the dependency information--building and traversing a tree shouldn't be that tough.... To preserve it, you'd only be able to parallelize the UTs within a module at a time (unless there's a different flag or something.) But it seems the key question is whether order can EVER be important for any reason. I for one would be willing to give up parallelization to get levelized tests. What are you seeing on your project? How do you allow tests to have dependencies and avoid order issues? Why is parallelization more important than that?
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 01 May 2014 14:40:41 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On 5/1/14, 2:28 PM, Jason Spencer wrote:
 But it seems the key question is whether order can EVER be
 important for any reason. I for one would be willing to give up
 parallelization to get levelized tests. What are you seeing on
 your project? How do you allow tests to have dependencies and
 avoid order issues? Why is parallelization more important than
 that?

I'll be blunt. What you say is technically sound (which is probably why you believe it is notable) but seems to me an unnecessarily complex engineering contraption that in all likelihood has more misuses than good uses. I fully understand you may think I'm a complete chowderhead for saying this; in the past I've been in your place and others have been in mine, and it took me years to appreciate both positions. -- Andrei

It's my understanding that given how druntime is put together, it should be possible to override some of its behaviors such that you could control the order in which tests were run (the main thing lacking at this point is the fact that you can currently only control it at module-level granularity) and that that's what existing third party unit test frameworks for D do. So, I would think that we could make it so that the default test runner does things the sensible way that works for most everyone, and then anyone who really wants more control can choose to override the normal test runner to do run the tests the way that they want to. That should be essentially the way that it is now. The main question then is which features we think are sensible for everyone, and I think that based on this discussion, at this point, it's primarily 1. Make it possible for druntime to access unit test functions individually. 2. Make it so that druntime runs unit test functions in parallel unless they're marked as needing to be run in serial (probably with a UDA for that purpose). 3. Make it so that we can name unittest blocks so that stack traces have better function names in them. With those sorted out, we can look at further features like whether we want to be able to run unit tests by name (or whatever other nice features we can come up with), but we might as well start there rather than trying to come up with a comprehensive list of the features that D's unit testing facilities should have (especially since we really should be erring on the side of simple). - Jonathan M Davis
May 01 2014
prev sibling next sibling parent "Jason Spencer" <j8spencer gmail.com> writes:
On Thursday, 1 May 2014 at 21:40:38 UTC, Andrei Alexandrescu 
wrote:
 I'll be blunt. What you say is technically sound (which is 
 probably why you believe it is notable)...

Well, I suppose that's not the MOST insulting brush-off I could hope for, but it falls short of encouraging me to contribute ideas for the improvement of the language. I'll just add this: I happen to introduce a colleague to the D webpage the other day, and ran across this in the overview: "D ... doesn't come with a VM, a religion, or an overriding philosophy. It's a practical language for practical programmers who need to get the job done quickly, reliably, and leave behind maintainable, easy to understand code." This business that only inherently parallel tests that never access disk, share setup, etc. are TRUE unit tests smack much more of religion than pragmatism. Indeed, phobos demonstrates that sometimes, the practical thing to do is to violate these normally good rules. Another overriding principle of D is that the easy thing to do should be the safe thing to do, and dangerous things should take some work. I don't see that reflected in the proposal to turn parallelism on by default. This seems like a time bomb waiting to go off on unsuspecting acolytes of the cult of inherently-parallel-tests-onlyism. If we don't want to consider how we can accommodate both camps here, then I must at least support Jonathan's modest suggestion that parallel UTs require active engagement rather than being the default.
May 01 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 2 May 2014 at 03:04:39 UTC, Jason Spencer wrote:
 If we don't want to consider how we can accommodate both camps 
 here, then I must at least support Jonathan's modest suggestion 
 that parallel UTs require active engagement rather than being 
 the default.

Use chroot() and fork(). Solves all problems.
May 01 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Friday, 2 May 2014 at 04:28:26 UTC, Ola Fosheim Grøstad wrote:
 On Friday, 2 May 2014 at 03:04:39 UTC, Jason Spencer wrote:
 If we don't want to consider how we can accommodate both camps 
 here, then I must at least support Jonathan's modest 
 suggestion that parallel UTs require active engagement rather 
 than being the default.

Use chroot() and fork(). Solves all problems.

You know, executing batches of tests in multiple processes could be a good compromise. You might still run into filesystem issues, but if you run a series of tests with a number of processes at the same time, you can at least guarantee that you won't run into shared memory issues.
May 01 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 2 May 2014 at 06:57:46 UTC, w0rp wrote:
 You know, executing batches of tests in multiple processes 
 could be a good compromise. You might still run into filesystem 
 issues, but if you run a series of tests with a number of 
 processes at the same time, you can at least guarantee that you 
 won't run into shared memory issues.

Using fork() would be good for multi-threaded unit testing or when testing global datastructures (singeltons). I don't get the desire for demanding that unit tests are "pure". That would miss the units that are most likely to blow up in an application. If you fork before opening any files it probably will work out ok.
May 02 2014
prev sibling parent "Jason Spencer" <j8spencer gmail.com> writes:
On Friday, 2 May 2014 at 14:59:50 UTC, Andrei Alexandrescu wrote:
 I need to make an amend to this because indeed it's more than 2 
 std deviations away from niceness: I have a long history of 
 ideas with a poor complexity/usefulness ratio, and I now wish 
 I'd received such a jolt. -- Andrei

I appreciate that, and can accept it in the spirit of mentoring and helpfulness. What might work even better for me, though, is to forego the assumption that I need such a jolt or that you are the person, in this forum at least, to provide it and simply address the merits or lack thereof of the suggestion as made. If we can't agree that a method, direct or indirect, to control the order of UTs is appropriate, then we should opt for the status quo. By my reading of this thread, that leaves us with no consensus that UTs MUST be order-independent, but that being able to parallelize is a good thing. It seems we can: 1. leave defaults as they are and make parallelization an option, or 2. make it the language model and allow people to dissent with an option I can agree with Andre that you'd rather have a solid, well-defined language that works in most cases without too many buttons, switches, and levers. I'm just not sure that jives with "easiest is safest" and "don't impose a model, provide a tool." To me, improving the performance of a non-performance-critical aspect does not weigh enough to counterbalance a safety risk and model imposition. How about others? Test names seems pretty much agreed to. I think the idea of making everything available to the druntime would let pretty much anyone do what they need.
May 02 2014
prev sibling next sibling parent reply "QAston" <qaston gmail.com> writes:
On Wednesday, 30 April 2014 at 15:43:35 UTC, Andrei Alexandrescu 
wrote:
 Hello,


 A coworker mentioned the idea that unittests could be run in 
 parallel (using e.g. a thread pool). I've rigged things to run 
 in parallel unittests across modules, and that works well. 
 However, this is too coarse-grained - it would be great if each 
 unittest could be pooled across the thread pool. That's more 
 difficult to implement.

 This brings up the issue of naming unittests. It's becoming 
 increasingly obvious that anonymous unittests don't quite scale 
 - coworkers are increasingly talking about "the unittest at 
 line 2035 is failing" and such. With unittests executing in 
 multiple threads and issuing e.g. logging output, this is only 
 likely to become more exacerbated. We've resisted named 
 unittests but I think there's enough evidence to make the 
 change.

 Last but not least, virtually nobody I know runs unittests and 
 then main. This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
    ...
 }

 I think it's time to change that. We could do it the 
 non-backward-compatible way by redefining -unittest to instruct 
 the compiler to not run main. Or we could define another flag 
 such as -unittest-only and then deprecate the existing one.

 Thoughts? Would anyone want to work on such stuff?


 Andrei

An existing library implementation: https://github.com/atilaneves/unit-threaded
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 9:24 AM, QAston wrote:
 An existing library implementation:
 https://github.com/atilaneves/unit-threaded

Nice! The "Warning: With dmd 2.064.2 and the gold linker on Linux 64-bit this code crashes." is hardly motivating though :o). I think this project is a confluence of a couple others, such as logging and a collection of specialized assertions. But it's hard to tell without documentation, and the linked output https://github.com/atilaneves/unit-threaded/blob/master/unit_threaded/io.d does not exist. Andrei
Apr 30 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 30 Apr 2014 09:28:18 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 I also wonder if its just better to extend/expose the unittest API
 for more advanced things like order of execution, test reporting,
 and parallel execution. And we can just support an external
 unittesting library to do all the advanced testing options.  

That would be pretty rad.

We can kinda do that. I guess the main problem for a simple approach is that unittests are functions but advanced frameworks often have unittest classes/objects. We can't really emulate that on top of functions. What we can easily do is parse UDAs on unittests and provide access to these UDAs. For example: ------------ module my.testlib; struct Author { string _name; string serialize() {return _name;} //Must be evaluated in CTFE } ------------ module test; import my.testlib; Author("The Author") unittest { //Code goes here } ------------ Then with the mentioned pull request we just add another field to the runtime unittest information struct: An associative array with string keys matching the qualified name of the UDA and as values the strings returned by serialize() (evaluated by CTFE). Then we have for the test runner: ------------ foreach( m; ModuleInfo ) { foreach(test; m.unitTests) { if("my.testlib.Author" in test.uda) writefln("Author: %s", test.uda["my.testlib.Author"]); } } ------------ This is some more work to implement though, but it's additive so we can first implement the basic mechanism in pull #1131 then add this uda stuff later.
Apr 30 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 30 Apr 2014 09:02:54 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 https://github.com/D-Programming-Language/dmd/pull/1131
 https://github.com/D-Programming-Language/druntime/pull/308

 To summarize: It provides a function pointer for every  unit test to
 druntime or user code. This is actually easy to do. Naming tests
 requires changes in the parser, but I guess that shouldn't be
 difficult either.  

That's fantastic, would you be willing to reconsider that work?

Sure, I'll have a look later today.
Apr 30 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 08:43:31AM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 Last but not least, virtually nobody I know runs unittests and then
 main.  This is quickly becoming an idiom:
 
 version(unittest) void main() {}
 else void main()
 {
    ...
 }
 
 I think it's time to change that. We could do it the
 non-backward-compatible way by redefining -unittest to instruct the
 compiler to not run main. Or we could define another flag such as
 -unittest-only and then deprecate the existing one.

Actually, I still run unittests before main. :) When I want to *not* run unittests, I just recompile with -release (and no -unittest). The nice thing about unittests running before main is that during the code-compile-test cycle I can have the unittests run *and* manually test the program afterwards -- usually in this case I only run the program once before modifying the code and recompiling, so it would be needless work to have to compile the program twice (once for unittests, once for main). An alternative, perhaps nicer, idea is to have a *runtime* switch to run unittests, recognized by druntime, perhaps something like: ./program --pragma-druntime-run-unittests Or something similarly unlikely to clash with real options accepted by the program. T -- Genius may have its limitations, but stupidity is not thus handicapped. -- Elbert Hubbard
Apr 30 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 30 April 2014 at 16:24:16 UTC, QAston wrote:
 On Wednesday, 30 April 2014 at 15:43:35 UTC, Andrei 
 Alexandrescu wrote:
 Hello,


 A coworker mentioned the idea that unittests could be run in 
 parallel (using e.g. a thread pool). I've rigged things to run 
 in parallel unittests across modules, and that works well. 
 However, this is too coarse-grained - it would be great if 
 each unittest could be pooled across the thread pool. That's 
 more difficult to implement.

 This brings up the issue of naming unittests. It's becoming 
 increasingly obvious that anonymous unittests don't quite 
 scale - coworkers are increasingly talking about "the unittest 
 at line 2035 is failing" and such. With unittests executing in 
 multiple threads and issuing e.g. logging output, this is only 
 likely to become more exacerbated. We've resisted named 
 unittests but I think there's enough evidence to make the 
 change.

 Last but not least, virtually nobody I know runs unittests and 
 then main. This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
   ...
 }

 I think it's time to change that. We could do it the 
 non-backward-compatible way by redefining -unittest to 
 instruct the compiler to not run main. Or we could define 
 another flag such as -unittest-only and then deprecate the 
 existing one.

 Thoughts? Would anyone want to work on such stuff?


 Andrei

An existing library implementation: https://github.com/atilaneves/unit-threaded

Beat me to it! :P The concurrency and naming aspects are exactly what drove me to write unit-threaded to begin with. I probably wouldn't have bothered if D already had the functionality I wanted. Atila
Apr 30 2014
prev sibling next sibling parent Orvid King via Digitalmars-d <digitalmars-d puremagic.com> writes:
As a note, I'm one of those that have used the main function in
addition to unittests. I use it in the unittest build mode of my JSON
serialization library, using the unittests to ensure I didn't break
anything, and then using the main to run a performance test that my
changes actually did make it faster.

On 4/30/14, H. S. Teoh via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Wed, Apr 30, 2014 at 08:43:31AM -0700, Andrei Alexandrescu via
 Digitalmars-d wrote:
 [...]
 Last but not least, virtually nobody I know runs unittests and then
 main.  This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
    ...
 }

 I think it's time to change that. We could do it the
 non-backward-compatible way by redefining -unittest to instruct the
 compiler to not run main. Or we could define another flag such as
 -unittest-only and then deprecate the existing one.

Actually, I still run unittests before main. :) When I want to *not* run unittests, I just recompile with -release (and no -unittest). The nice thing about unittests running before main is that during the code-compile-test cycle I can have the unittests run *and* manually test the program afterwards -- usually in this case I only run the program once before modifying the code and recompiling, so it would be needless work to have to compile the program twice (once for unittests, once for main). An alternative, perhaps nicer, idea is to have a *runtime* switch to run unittests, recognized by druntime, perhaps something like: ./program --pragma-druntime-run-unittests Or something similarly unlikely to clash with real options accepted by the program. T -- Genius may have its limitations, but stupidity is not thus handicapped. -- Elbert Hubbard

Apr 30 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 30 April 2014 at 16:32:19 UTC, Andrei Alexandrescu 
wrote:
 On 4/30/14, 9:24 AM, QAston wrote:
 An existing library implementation:
 https://github.com/atilaneves/unit-threaded

Nice! The "Warning: With dmd 2.064.2 and the gold linker on Linux 64-bit this code crashes." is hardly motivating though :o). I think this project is a confluence of a couple others, such as logging and a collection of specialized assertions. But it's hard to tell without documentation, and the linked output https://github.com/atilaneves/unit-threaded/blob/master/unit_threaded/io.d does not exist. Andrei

I'm thinking of removing the warning but I have no idea how many people are using dmd 2.064.2, and it does crash if used with ld.gold. It was a dmd bug that got fixed (I voted on it). I fixed the Markdown link. That's what happens when I move code around! If you want to see what the output is like you can check out https://travis-ci.org/atilaneves/cerealed or git clone it and run dub test. I think seeing failing output is just as interesting as well, so there's a failing example in there that can be executed. The README.md says how. When I first wrote this I tried using it on Phobos to see if I could run the unit tests a lot faster but didn't have a lot of luck. I think I ran out of memory trying to reflect on its modules but I can't remember. I should try that again. Atila
Apr 30 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
I believe only missing step right now is propagation of UDA's to 
RTInfo when demanded. Everything else can be done as Phobos 
solution.

And if requirement to have all modules transitively accessible 
from root one is acceptable it can be already done with 
http://dlang.org/traits.html#getUnitTests

Simplicity of D unit tests is their best feature.
Apr 30 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-04-30 19:30, Dicebot wrote:
 I believe only missing step right now is propagation of UDA's to RTInfo
 when demanded. Everything else can be done as Phobos solution.

I don't see why this is necessary for this case. -- /Jacob Carlborg
Apr 30 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-05-01 09:10, Dicebot wrote:

 It is not strictly necessary but you can't reliably get all unit test
 blocks during compile-time (must be transitively imported) and current
 run-time reflection for tests is missing any data but actual function
 pointers. I am personally perfectly satisfied with "single root module
 imports all" approach but it is likely to be complained if proposed as
 "standard" way.

The current runner wouldn't work. It needs to be replace with one that uses __traits(getUnitTests). -- /Jacob Carlborg
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 08:59:42 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

In general, I agree. In reality, there are times when having state across unit tests makes sense - especially when there's expensive setup required for the tests. While it's not something that I generally like to do, I know that we have instances of that where I work. Also, if the unit tests have to deal with shared resources, they may very well be theoretically independent but would run afoul of each other if run at the same time - a prime example of this would be std.file, which has to operate on the file system. I fully expect that if std.file's unit tests were run in parallel, they would break. Unit tests involving sockets would be another type of test which would be at high risk of breaking, depending on what sockets they need. Honestly, the idea of running unit tests in parallel makes me very nervous. In general, across modules, I'd expect it to work, but there will be occasional cases where it will break. Across the unittest blocks in a single module, I'd be _very_ worried about breakage. There is nothing whatsoever in the language which guarantees that running them in parallel will work or even makes sense. All that protects us is the convention that unit tests are usually independent of each other, and in my experience, it's common enough that they're not independent that I think that blindly enabling parallelization of unit tests across a single module is definitely a bad idea. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 30 April 2014 at 17:50:34 UTC, Jonathan M Davis via 
Digitalmars-d wrote:
 On Wed, 30 Apr 2014 08:59:42 -0700
 Andrei Alexandrescu via Digitalmars-d 
 <digitalmars-d puremagic.com>
 wrote:

 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

In general, I agree. In reality, there are times when having state across unit tests makes sense - especially when there's expensive setup required for the tests. While it's not something that I generally like to do, I know that we have instances of that where I work. Also, if the unit tests have to deal with shared resources, they may very well be theoretically independent but would run afoul of each other if run at the same time - a prime example of this would be std.file, which has to operate on the file system. I fully expect that if std.file's unit tests were run in parallel, they would break. Unit tests involving sockets would be another type of test which would be at high risk of breaking, depending on what sockets they need. Honestly, the idea of running unit tests in parallel makes me very nervous. In general, across modules, I'd expect it to work, but there will be occasional cases where it will break. Across the unittest blocks in a single module, I'd be _very_ worried about breakage. There is nothing whatsoever in the language which guarantees that running them in parallel will work or even makes sense. All that protects us is the convention that unit tests are usually independent of each other, and in my experience, it's common enough that they're not independent that I think that blindly enabling parallelization of unit tests across a single module is definitely a bad idea. - Jonathan M Davis

You're right; blindly enabling parallelisation after the fact is likely to cause problems. Unit tests though, by definition (and I'm aware there are more than one) have to be independent. Have to not touch the filesystem, or the network. Only CPU and RAM. In my case, and since I had the luxury of implementing a framework first and only writing tests after it was done, running them in parallel was an extra check that they are in fact independent. Now, it does happen that you're testing code that isn't thread-safe itself, and yes, in that case you have to run them in a single thread. That's why I added the SingleThreaded UDA to my library to enable that. As soon as I tried calling legacy C code... We could always make running in threads opt-in. Atila
Apr 30 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 30 April 2014 at 17:30:30 UTC, Dicebot wrote:
 I believe only missing step right now is propagation of UDA's 
 to RTInfo when demanded. Everything else can be done as Phobos 
 solution.

 And if requirement to have all modules transitively accessible 
 from root one is acceptable it can be already done with 
 http://dlang.org/traits.html#getUnitTests

 Simplicity of D unit tests is their best feature.

IMHO this best feature is only useful when writing a small script-like program. The hassle of using anything more heavy-duty is likely to make one not want to write tests. The unittest blocks are simple, and that's good. But for me I wouldn't (and haven't) use them for "real work". When tests pass, it doesn't really matter if they were written with only using assert or what the output was like or any of those things. But when they fail, I want to: . Run the failing test(s) in isolation, selecting them on the command-line by name . Have tests grouped in categories (I use packages) to run similar tests together . Be able to enable debug output that is normally supressed . Know the name of the test to know which one is failing . Have meaningful output from the failure without having to construct said meaningful output myself (assertEquals vs assert) I don't know about anyone else, but I make my tests fail a lot. I also added threading, hidden tests, and tests expected to fail to that list but they are nice-to-have features. I can't do without the rest though. Also, I like pretty colours in the output for failure and success, but that might be just me :P Atila
Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 17:58:34 +0000
Atila Neves via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 Unit tests though, by definition (and I'm aware there are more 
 than one) have to be independent. Have to not touch the 
 filesystem, or the network. Only CPU and RAM.

I disagree with this. A unit test is a test that tests a single piece of functionality - generally a function - and there are functions which have to access the file system or network. And those tests are done in unittest blocks just like any other unit test. I would very much consider std.file's tests to be unit tests. But even if you don't want to call them unit tests, because they access the file system, the reality of the matter is that tests like them are going to be run in unittest blocks, and we have to take that into account when we decide how we want unittest blocks to be run (e.g. whether they're parallelizable or not). - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 30 April 2014 at 15:54:42 UTC, bearophile wrote:
 We've resisted named unittests but I think there's enough
 evidence to make the change.

Yes, the optional name for unittests is an improvement: unittest {} unittest foo {} I am very glad your coworker find such usability problems :-)

If we do "name" the unittests, then can we name them with strings? No need to polute namespace with ugly symbols. Also: //---- unittest "Sort: Non-Lvalue RA range" { ... } //---- vs //---- unittest SortNonLvalueRARange { ... } //----
Apr 30 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
monarch_dodra:

 If we do "name" the unittests, then can we name them with 
 strings? No need to polute namespace with ugly symbols.

Are UDAs enough? uname("foo") unittest {} What I'd like is to tie one or more unittests to other entities, like all the unittests of a specific function. Bye, bearophile
Apr 30 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 30 April 2014 at 18:04:43 UTC, Atila Neves wrote:
 On Wednesday, 30 April 2014 at 17:30:30 UTC, Dicebot wrote:
 I believe only missing step right now is propagation of UDA's 
 to RTInfo when demanded. Everything else can be done as Phobos 
 solution.

 And if requirement to have all modules transitively accessible 
 from root one is acceptable it can be already done with 
 http://dlang.org/traits.html#getUnitTests

 Simplicity of D unit tests is their best feature.

IMHO this best feature is only useful when writing a small script-like program. The hassle of using anything more heavy-duty is likely to make one not want to write tests. The unittest blocks are simple, and that's good. But for me I wouldn't (and haven't) use them for "real work". When tests pass, it doesn't really matter if they were written with only using assert or what the output was like or any of those things. But when they fail, I want to: . Run the failing test(s) in isolation, selecting them on the command-line by name . Have tests grouped in categories (I use packages) to run similar tests together . Be able to enable debug output that is normally supressed . Know the name of the test to know which one is failing . Have meaningful output from the failure without having to construct said meaningful output myself (assertEquals vs assert) I don't know about anyone else, but I make my tests fail a lot.

I think this is key difference. For me failing unit test is always exceptional situation. And if test group is complex enough to require categorization then either my code is not procedural enough or module is just too big and needs to be split. There are of course always some tests with complicated environment and/or I/O. But those are never unit tests and thus part of completely different framework.
Apr 30 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 30 April 2014 at 18:19:34 UTC, Jonathan M Davis via 
Digitalmars-d wrote:
 On Wed, 30 Apr 2014 17:58:34 +0000
 Atila Neves via Digitalmars-d <digitalmars-d puremagic.com> 
 wrote:
 Unit tests though, by definition (and I'm aware there are more 
 than one) have to be independent. Have to not touch the 
 filesystem, or the network. Only CPU and RAM.

I disagree with this. A unit test is a test that tests a single piece of functionality - generally a function - and there are functions which have to access the file system or network.

They _use_ access to file system or network, but it is _not_ their functionality. Unit testing is all about verifying small perfectly separated pieces of functionality which don't depend on correctness / stability of any other functions / programs. Doing I/O goes against it pretty much by definition and is unfortunately one of most common testing antipatterns.
Apr 30 2014
prev sibling next sibling parent Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 4/30/14, bearophile via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 What I'd like is to tie one or more unittests to other entities,
 like all the unittests of a specific function.

This would also lead to a more stable documented unittest feature and the ability to declare documented unittests outside the scope of the target symbol. E.g. if you have a templated aggregate and a function inside it you may want to add a single documented unittest for the function /outside/ the aggregate, otherwise it will get compiled for every unique instance of that aggregate.
Apr 30 2014
prev sibling next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 2014-04-30 at 10:50 -0700, Jonathan M Davis via Digitalmars-d
wrote:
[…]
 In general, I agree. In reality, there are times when having state
 across unit tests makes sense - especially when there's expensive setup
 required for the tests. While it's not something that I generally
 like to do, I know that we have instances of that where I work. Also, if
 the unit tests have to deal with shared resources, they may very well be
 theoretically independent but would run afoul of each other if run at
 the same time - a prime example of this would be std.file, which has to
 operate on the file system. I fully expect that if std.file's unit
 tests were run in parallel, they would break. Unit tests involving
 sockets would be another type of test which would be at high risk of
 breaking, depending on what sockets they need.

Surely if there is expensive set up you are doing an integration or system test not a unit test. In a unit test all expensive set up should be mocked out.
 Honestly, the idea of running unit tests in parallel makes me very
 nervous. In general, across modules, I'd expect it to work, but there
 will be occasional cases where it will break. Across the unittest
 blocks in a single module, I'd be _very_ worried about breakage. There
 is nothing whatsoever in the language which guarantees that running
 them in parallel will work or even makes sense. All that protects us is
 the convention that unit tests are usually independent of each other,
 and in my experience, it's common enough that they're not independent
 that I think that blindly enabling parallelization of unit tests across
 a single module is definitely a bad idea.

All tests should be independent, therefore there should be no problem executing all tests at the same time and/or in any order. If tests have to be executed in a specific order then they are not separate tests and should be merged into a single test — which likely means they are integration or system tests not unit tests. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 30 2014
prev sibling next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 2014-04-30 at 11:19 -0700, Jonathan M Davis via Digitalmars-d
wrote:
[…]
 I disagree with this. A unit test is a test that tests a single piece
 of functionality - generally a function - and there are functions which
 have to access the file system or network. And those tests are done in

These are integration/system tests not unit tests. For unit tests network activity should be mocked out.
 unittest blocks just like any other unit test. I would very much
 consider std.file's tests to be unit tests. But even if you don't
 want to call them unit tests, because they access the file system, the
 reality of the matter is that tests like them are going to be run in
 unittest blocks, and we have to take that into account when we decide
 how we want unittest blocks to be run (e.g. whether they're
 parallelizable or not).

In which case D is wrong to allow them in the unittest blocks and should introduce a new way of handling these tests. And even then all tests can and should be parallelized. If they cannot be then there is an inappropriate dependency. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 30 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-04-30 17:43, Andrei Alexandrescu wrote:
 Hello,


 A coworker mentioned the idea that unittests could be run in parallel
 (using e.g. a thread pool). I've rigged things to run in parallel
 unittests across modules, and that works well. However, this is too
 coarse-grained - it would be great if each unittest could be pooled
 across the thread pool. That's more difficult to implement.

Can't we just collect all unit tests with __traits(getUnitTests) and put them through std.parallelism: foreach (unitTest ; unitTests.parallel) unitTest();
 This brings up the issue of naming unittests. It's becoming increasingly
 obvious that anonymous unittests don't quite scale - coworkers are
 increasingly talking about "the unittest at line 2035 is failing" and
 such. With unittests executing in multiple threads and issuing e.g.
 logging output, this is only likely to become more exacerbated. We've
 resisted named unittests but I think there's enough evidence to make the
 change.

Named unit tests are already possible with the help of UDA's: name("foo bar") unittest { assert(true); } I've tried several times here, in reviews, to get people to add some description to the unit tests. But so far no one has agreed. I'm using something quite similar to RSpec from the Ruby world: describe! "toMsec" in { it! "returns the time in milliseconds" in { assert(true); } } This uses the old syntax, with UDA's it becomes something like this: describe("toMsec") { it("returns the time in milliseconds") unittest { assert(true); } }
 Last but not least, virtually nobody I know runs unittests and then
 main. This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

Or "dmd -unittest -main -run foo.d"
 I think it's time to change that. We could do it the
 non-backward-compatible way by redefining -unittest to instruct the
 compiler to not run main. Or we could define another flag such as
 -unittest-only and then deprecate the existing one.

Fine by me, I don't like that "main" is run after the unit tests.
 Thoughts? Would anyone want to work on such stuff?

Are you thinking of built-in support or an external library? -- /Jacob Carlborg
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:19 PM, Jacob Carlborg wrote:
 On 2014-04-30 17:43, Andrei Alexandrescu wrote:
 Hello,


 A coworker mentioned the idea that unittests could be run in parallel
 (using e.g. a thread pool). I've rigged things to run in parallel
 unittests across modules, and that works well. However, this is too
 coarse-grained - it would be great if each unittest could be pooled
 across the thread pool. That's more difficult to implement.

Can't we just collect all unit tests with __traits(getUnitTests) and put them through std.parallelism: foreach (unitTest ; unitTests.parallel) unitTest();

I didn't know of that trait; I adapted code from druntime/src/test_runner.d.
 Named unit tests are already possible with the help of UDA's:

  name("foo bar") unittest
 {
      assert(true);
 }

 I've tried several times here, in reviews, to get people to add some
 description to the unit tests. But so far no one has agreed.

Yah I think that's possible but I'd like the name to be part of the function name as well e.g. unittest__%s.
 I'm using something quite similar to RSpec from the Ruby world:

 describe! "toMsec" in {
      it! "returns the time in milliseconds" in {
          assert(true);
      }
 }

 This uses the old syntax, with UDA's it becomes something like this:

  describe("toMsec")
 {
       it("returns the time in milliseconds") unittest
      {
          assert(true);
      }
 }

That looks... interesting.
 Thoughts? Would anyone want to work on such stuff?

Are you thinking of built-in support or an external library?

Built in with possible help from druntime and/or std. Andrei
Apr 30 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-04-30 22:41, Andrei Alexandrescu wrote:

 Yah I think that's possible but I'd like the name to be part of the
 function name as well e.g. unittest__%s.

Why is that necessary? To have the correct symbol name when debugging?
 I'm using something quite similar to RSpec from the Ruby world:

 describe! "toMsec" in {
      it! "returns the time in milliseconds" in {
          assert(true);
      }
 }

 This uses the old syntax, with UDA's it becomes something like this:

  describe("toMsec")
 {
       it("returns the time in milliseconds") unittest
      {
          assert(true);
      }
 }

That looks... interesting.

The Ruby syntax looks like this: describe "toMsec" do it "reruns the time in milliseconds" do assert true end context "when the time parameter is nil" do it "returns nil" do assert true end end end The interesting part about the Ruby implementation is that each it-block (the code between do/end) is executed in the context of an anonymous class instance. Each describe/context-block is turned in to a class, nested blocks inherit from the outer block. In D the implementation would look like this: class __toMsec { void __someUniqueName123 () { } class __SomeUniqueName456 : __toMsec { void __someUniqueName789 () { } } } Each it-block (unit test) will be executed in a new instance of the closest surrounding class. This means you can have helper methods and instance variables shared across multiple test, but they will each get a fresh copy of the data. Since the describe-blocks are implemented with classes that inherit you can override helper methods in subclasses. The unit test runner can also print out a documentation, basically all text in the "it" and "describe" parameters. Something like this: https://coderwall-assets-0.s3.amazonaws.com/uploads/picture/file/1949/rspec_html_screen.png -- /Jacob Carlborg
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 4:41 AM, Jacob Carlborg wrote:
 On 2014-04-30 22:41, Andrei Alexandrescu wrote:

 Yah I think that's possible but I'd like the name to be part of the
 function name as well e.g. unittest__%s.

Why is that necessary? To have the correct symbol name when debugging?

It's nice to have the name available in other tools (stack trace, debugger).
 The Ruby syntax looks like this:

 The unit test runner can also print out a documentation, basically all
 text in the "it" and "describe" parameters. Something like this:
 https://coderwall-assets-0.s3.amazonaws.com/uploads/picture/file/1949/rspec_html_screen.png

That's all nice, but I feel we're going gung ho with overengineering already. If we give unittests names and then offer people a button "parallelize unittests" to push (don't even specify the number of threads! let the system figure it out depending on cores), that's a good step to a better world. Andrei
May 01 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-01 17:15, Andrei Alexandrescu wrote:

 That's all nice, but I feel we're going gung ho with overengineering
 already. If we give unittests names and then offer people a button
 "parallelize unittests" to push (don't even specify the number of
 threads! let the system figure it out depending on cores), that's a good
 step to a better world.

Sure. But on the other hand, why should D not have a great unit testing framework built-in. -- /Jacob Carlborg
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 11:49 AM, Jacob Carlborg wrote:
 On 2014-05-01 17:15, Andrei Alexandrescu wrote:

 That's all nice, but I feel we're going gung ho with overengineering
 already. If we give unittests names and then offer people a button
 "parallelize unittests" to push (don't even specify the number of
 threads! let the system figure it out depending on cores), that's a good
 step to a better world.

Sure. But on the other hand, why should D not have a great unit testing framework built-in.

It should. My focus is to get (a) unittest names and (b) parallel testing into the language ASAP. Andrei
May 01 2014
next sibling parent Ary Borenszweig <ary esperanto.org.ar> writes:
On 5/1/14, 4:22 PM, Andrei Alexandrescu wrote:
 On 5/1/14, 11:49 AM, Jacob Carlborg wrote:
 On 2014-05-01 17:15, Andrei Alexandrescu wrote:

 That's all nice, but I feel we're going gung ho with overengineering
 already. If we give unittests names and then offer people a button
 "parallelize unittests" to push (don't even specify the number of
 threads! let the system figure it out depending on cores), that's a good
 step to a better world.

Sure. But on the other hand, why should D not have a great unit testing framework built-in.

It should. My focus is to get (a) unittest names and (b) parallel testing into the language ASAP. Andrei

What's the rush?
May 01 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 8:16 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 19:22:36 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 11:49 AM, Jacob Carlborg wrote:
 On 2014-05-01 17:15, Andrei Alexandrescu wrote:

 That's all nice, but I feel we're going gung ho with overengineering
 already. If we give unittests names and then offer people a button
 "parallelize unittests" to push (don't even specify the number of
 threads! let the system figure it out depending on cores), that's a
 good
 step to a better world.

Sure. But on the other hand, why should D not have a great unit testing framework built-in.

It should. My focus is to get (a) unittest names and (b) parallel testing into the language ASAP. Andrei

It is wrong approach. Proper one is to be able to define any sort of test running system in library code while still being 100% compatible with naive `dmd -unittest`. We are almost quite there, only step missing is transferring attributes to runtime unittest block reflection.

Penalizing unittests that were bad in the first place is pretty attractive, but propagating attributes properly is even better. -- Andrei
May 05 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote:
 Am Wed, 30 Apr 2014 22:19:24 +0200
 schrieb Jacob Carlborg <doob me.com>:

 __traits(getUnitTests)

I've always wondered, but never asked: Doesn't __traits(getUnitTests) usage suffer from the same problem std.benchmark had? That in order to make it work for all modules you a) have to explicitly mention every module b) use module constructors, leading to module constructor dependency hell (afaik the main reason we don't have a std.benchmark now) ?

You only need to make sure all modules are transitively imported from initial one. Everything else can be done via recursive reflection with __traits(allMembers), module constructors are not needed either.
May 01 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-01 11:37, Dicebot wrote:

 You only need to make sure all modules are transitively imported from
 initial one.

The solution for that would be RMInfo [1], like RTInfo but for modules instead of types. [1] https://github.com/D-Programming-Language/dmd/pull/2271 -- /Jacob Carlborg
May 01 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-05-01 14:00, Johannes Pfau wrote:

 But on a quick look I don't understand how this (or DRT #775) completely
 solves the issue.

 Now you don't have to import the modules anymore, which is a step
 forward, but let's say I want to used this to find all SubClasses of a
 class.

 Now I can inspect that code in CTFE, but how do I build a list of all
 subclasses? In the most generic way (with DLLs/shared libraries) this
 can only work if code can be executed at runtime, so we'd need a way to
 emit module constructors AFAICS. These should be able to avoid the
 usual constructor dependency rules though.

RMInfo only helps with all the modules the compiler sees during a given compile run. -- /Jacob Carlborg
May 01 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote:
 Am Wed, 30 Apr 2014 22:19:24 +0200
 schrieb Jacob Carlborg <doob me.com>:

 __traits(getUnitTests)

I've always wondered, but never asked: Doesn't __traits(getUnitTests) usage suffer from the same problem std.benchmark had? That in order to make it work for all modules you a) have to explicitly mention every module b) use module constructors, leading to module constructor dependency hell (afaik the main reason we don't have a std.benchmark now) ?

You only need to make sure all modules are transitively imported from initial one. Everything else can be done via recursive reflection with __traits(allMembers), module constructors are not needed either. One can also generate special test entry module that imports all project modules using build system help.
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 4:31 AM, Johannes Pfau wrote:
  Andrei do you think having to explicitly import modules to be tested
 is an issue?

Well it kinda is. All that's written on the package is "unittest", we should add no fine print to it. -- Andrei
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 12:05 PM, Johannes Pfau wrote:
 Am Thu, 01 May 2014 08:07:51 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 5/1/14, 4:31 AM, Johannes Pfau wrote:
  Andrei do you think having to explicitly import modules to be
 tested is an issue?

Well it kinda is. All that's written on the package is "unittest", we should add no fine print to it. -- Andrei

It'd be possible to make it work transparently, but that's much more work. We might need to do that at some point, as it's also necessary for std.benchmark and similar code but for now that's probably over-engineering. Here's the revived pull request: https://github.com/D-Programming-Language/dmd/pull/3518 https://github.com/D-Programming-Language/druntime/pull/782

I'm unclear what this work does even after having read the description. What does it require in addition to just sprinkling unittests around? -- Andrei
May 01 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 12:47 PM, Johannes Pfau wrote:
 Am Thu, 01 May 2014 12:26:07 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 5/1/14, 12:05 PM, Johannes Pfau wrote:
 Here's the revived pull request:
 https://github.com/D-Programming-Language/dmd/pull/3518
 https://github.com/D-Programming-Language/druntime/pull/782

I'm unclear what this work does even after having read the description. What does it require in addition to just sprinkling unittests around? -- Andrei

Nothing. This just changes the way druntime internally handles the unit tests, but nothing changes for the user:

Great, thanks. Just making sure there's no unstated assumptions somewhere. Help with reviewing from the compiler and druntime folks would be appreciated! -- Andrei
May 01 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 19:22:36 UTC, Andrei Alexandrescu 
wrote:
 On 5/1/14, 11:49 AM, Jacob Carlborg wrote:
 On 2014-05-01 17:15, Andrei Alexandrescu wrote:

 That's all nice, but I feel we're going gung ho with 
 overengineering
 already. If we give unittests names and then offer people a 
 button
 "parallelize unittests" to push (don't even specify the 
 number of
 threads! let the system figure it out depending on cores), 
 that's a good
 step to a better world.

Sure. But on the other hand, why should D not have a great unit testing framework built-in.

It should. My focus is to get (a) unittest names and (b) parallel testing into the language ASAP. Andrei

It is wrong approach. Proper one is to be able to define any sort of test running system in library code while still being 100% compatible with naive `dmd -unittest`. We are almost quite there, only step missing is transferring attributes to runtime unittest block reflection.
May 05 2014
prev sibling next sibling parent =?UTF-8?B?IsOBdGlsYQ==?= Neves " <atila.neves gmail.com> writes:
On Wednesday, 30 April 2014 at 19:20:20 UTC, Dicebot wrote:
 On Wednesday, 30 April 2014 at 18:04:43 UTC, Atila Neves wrote:
 On Wednesday, 30 April 2014 at 17:30:30 UTC, Dicebot wrote:
 I believe only missing step right now is propagation of UDA's 
 to RTInfo when demanded. Everything else can be done as 
 Phobos solution.

 And if requirement to have all modules transitively 
 accessible from root one is acceptable it can be already done 
 with http://dlang.org/traits.html#getUnitTests

 Simplicity of D unit tests is their best feature.

IMHO this best feature is only useful when writing a small script-like program. The hassle of using anything more heavy-duty is likely to make one not want to write tests. The unittest blocks are simple, and that's good. But for me I wouldn't (and haven't) use them for "real work". When tests pass, it doesn't really matter if they were written with only using assert or what the output was like or any of those things. But when they fail, I want to: . Run the failing test(s) in isolation, selecting them on the command-line by name . Have tests grouped in categories (I use packages) to run similar tests together . Be able to enable debug output that is normally supressed . Know the name of the test to know which one is failing . Have meaningful output from the failure without having to construct said meaningful output myself (assertEquals vs assert) I don't know about anyone else, but I make my tests fail a lot.

I think this is key difference. For me failing unit test is always exceptional situation.

I TDD a lot. Tests failing are normal. Not only that, I refactor a lot as well. Which causes tests to fail. Fortunately I have tests failing to tell me I screwed up. Even if failing tests were exceptional, I still want everything I just mentioned.
And if test group is complex
 enough to require categorization then either my code is not 
 procedural enough or module is just too big and needs to be 
 split.

And when I split them I put them into a subcategory.
Apr 30 2014
prev sibling next sibling parent =?UTF-8?B?IsOBdGlsYQ==?= Neves" <atila.neves gmail.com> writes:
On Wednesday, 30 April 2014 at 18:19:34 UTC, Jonathan M Davis via 
Digitalmars-d wrote:
 On Wed, 30 Apr 2014 17:58:34 +0000
 Atila Neves via Digitalmars-d <digitalmars-d puremagic.com> 
 wrote:
 Unit tests though, by definition (and I'm aware there are more 
 than one) have to be independent. Have to not touch the 
 filesystem, or the network. Only CPU and RAM.

I disagree with this. A unit test is a test that tests a single piece of functionality - generally a function - and there are functions which have to access the file system or network. And those tests are done in unittest blocks just like any other unit test. I would very much consider std.file's tests to be unit tests. But even if you don't want to call them unit tests, because they access the file system, the reality of the matter is that tests like them are going to be run in unittest blocks, and we have to take that into account when we decide how we want unittest blocks to be run (e.g. whether they're parallelizable or not). - Jonathan M Davis

On what's a unit test: I +1 everything Dicebot and Russell Winder said. Of course there are functions with side effects. Of course they should be tested. But those tests aren't unit tests. Which won't stop people from using a unit test framework to run them. In fact, every test I've ever written using python's unittest module was an integration test. But again, you're right. Whatever changes happen have to take into account the current status. And the current status makes it difficult if not impossible to run existing tests in multiple threads by default. One could argue that the Phobos tests should be changed too, but that won't help with the existing client codebase out there.
Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 13:26:40 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On 4/30/14, 10:50 AM, Jonathan M Davis via Digitalmars-d wrote:
 There
 is nothing whatsoever in the language which guarantees that running
 them in parallel will work or even makes sense.

Default thread-local globals? -- Andrei

Sure, that helps, but it's trivial to write a unittest block which depends on a previous unittest block, and as soon as a unittest block uses an external resource such as a socket or file, then even if a unittest block doesn't directly depend on the end state of a previous unittest block, it still depends on external state which could be affected by other unittest blocks. So, ultimately, the language really doesn't ensure that running a unittest block can be parallelized. If it's pure as bearophile suggested, then it can be done, but as long as a unittest block is impure, then it can rely on global state - even inadvertently - (be it state directly in the program or state outside the program) and therefore not work when pararellized. So, I suppose that you could parallelize unittest blocks if they were marked as pure (though I'm not sure if that's currently a legal thing to do), but impure unittest blocks aren't guaranteed to be parallelizable. I'm all for making it possible to parallelize unittest block execution, but as it stands, doing so automatically would be a bad idea. We could make it so that a unittest block could be marked as parallelizable, or we could even move towards making parallelizable the default and require that a unittest block be marked as unparallelizable, but we'd have to be very careful with that, as it will break code if we're not careful about how we do that transition. I'm inclined to think that marking unittest blocks as pure to parallelize them is a good idea, because then the unittest blocks that are guaranteed to be parallelizable are run in parallel, whereas those that aren't wouldn't be. The primary dowside would be that the cases where the programmer knew that they could be parallelized but they weren't pure, since those unittest blocks wouldn't be parallelized. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 18:53:22 +0000
monarch_dodra via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wednesday, 30 April 2014 at 15:54:42 UTC, bearophile wrote:
 We've resisted named unittests but I think there's enough
 evidence to make the change.

Yes, the optional name for unittests is an improvement: unittest {} unittest foo {} I am very glad your coworker find such usability problems :-)

If we do "name" the unittests, then can we name them with strings? No need to polute namespace with ugly symbols. Also: //---- unittest "Sort: Non-Lvalue RA range" { ... } //---- vs //---- unittest SortNonLvalueRARange { ... } //----

It would be simple enough to avoid polluting the namespace. IIRC, right now, the unittest blocks get named after the line number that they're on. All we'd have to do is change it so that their name included the name given by the programmer rather than being the name given by the programmer. e.g. unittest(testFoo) { } results in a function called something like unittest_testFoo. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 4/30/14, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 This brings up the issue of naming unittests.

See also this ER where I discuss why I wanted this recently: https://issues.dlang.org/show_bug.cgi?id=12473
Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 21:09:14 +0100
Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wed, 2014-04-30 at 11:19 -0700, Jonathan M Davis via Digitalmars-d
 wrote:
 unittest blocks just like any other unit test. I would very much
 consider std.file's tests to be unit tests. But even if you don't
 want to call them unit tests, because they access the file system,
 the reality of the matter is that tests like them are going to be
 run in unittest blocks, and we have to take that into account when
 we decide how we want unittest blocks to be run (e.g. whether
 they're parallelizable or not).

In which case D is wrong to allow them in the unittest blocks and should introduce a new way of handling these tests. And even then all tests can and should be parallelized. If they cannot be then there is an inappropriate dependency.

Why? Because Andrei suddenly proposed that we parallelize unittest blocks? If I want to test a function, I'm going to put a unittest block after it to test it. If that means accessing I/O, then it means accessing I/O. If that means messing with mutable, global variables, then that means messing with mutable, global variables. Why should I have to put the tests elsewhere or make is that they don't run whenthe -unttest flag is used just because they don't fall under your definition of "unit" test? There is nothing in the language which has ever mandated that unittest blocks be parallelizable or that they be pure (which is essentially what you're saying all unittest blocks should be). And restricting unittest blocks so that they have to be pure (be it conceptually pure or actually pure) would be a _loss_ of functionality. Sure, let's make it possible to parallelize unittest blocks where appropriate, but I contest that we should start requiring that unittest blocks be pure (which is what a function has to be in order to be pararellized whether it's actually marked as pure or not). That would force us to come up with some other testing mechanism to run those tests when there is no need to do so (and I would argue that there is no compelling reason to do so other than ideology with regards to what is truly a "unit" test). On the whole, I think that unittest blocks work very well as they are. If we want to expand on their features, then great, but let's do so without adding new restrictions to them. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 02:48:38PM -0700, Jonathan M Davis via Digitalmars-d
wrote:
 On Wed, 30 Apr 2014 21:09:14 +0100
 Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 In which case D is wrong to allow them in the unittest blocks and
 should introduce a new way of handling these tests. And even then
 all tests can and should be parallelized. If they cannot be then
 there is an inappropriate dependency.

Why? Because Andrei suddenly proposed that we parallelize unittest blocks? If I want to test a function, I'm going to put a unittest block after it to test it. If that means accessing I/O, then it means accessing I/O. If that means messing with mutable, global variables, then that means messing with mutable, global variables. Why should I have to put the tests elsewhere or make is that they don't run whenthe -unttest flag is used just because they don't fall under your definition of "unit" test?

What about allowing pure marking on unittests, and those unittests that are marked pure will be parallelized, and those that aren't marked will be run serially? T -- Amateurs built the Ark; professionals built the Titanic.
Apr 30 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 02:25:22PM -0700, Jonathan M Davis via Digitalmars-d
wrote:
[...]
 Sure, that helps, but it's trivial to write a unittest block which
 depends on a previous unittest block, and as soon as a unittest block
 uses an external resource such as a socket or file, then even if a
 unittest block doesn't directly depend on the end state of a previous
 unittest block, it still depends on external state which could be
 affected by other unittest blocks.

In this case I'd argue that the test was poorly-written. I can see multiple unittests using, say, the same temp filename for testing file I/O, in which case they shouldn't be parallelized; but if a unittest depends on a file created by a previous unittest, then something is very, very wrong with the unittest. [...]
 I'm inclined to think that marking unittest blocks as pure to
 parallelize them is a good idea, because then the unittest blocks that
 are guaranteed to be parallelizable are run in parallel, whereas those
 that aren't wouldn't be.

Agreed.
 The primary dowside would be that the cases where the programmer knew
 that they could be parallelized but they weren't pure, since those
 unittest blocks wouldn't be parallelized.

Is it a big loss to have *some* unittests non-parallelizable? (I don't know, do we have hard data on this front?) T -- The two rules of success: 1. Don't tell everything you know. -- YHL
Apr 30 2014
prev sibling next sibling parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
 What about allowing pure marking on unittests, and those 
 unittests that
 are marked pure will be parallelized, and those that aren't 
 marked will
 be run serially?

I guess that goes for inferred purity aswell...
Apr 30 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 Apr 2014 13:50:10 -0400, Jonathan M Davis via Digitalmars-d  
<digitalmars-d puremagic.com> wrote:

 On Wed, 30 Apr 2014 08:59:42 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

In general, I agree. In reality, there are times when having state across unit tests makes sense - especially when there's expensive setup required for the tests.

int a; unittest { // set up a; } unittest { // use a; } ==> unittest { int a; { // set up a; } { // use a; } } It makes no sense to do it the first way, you are not gaining anything.
 Honestly, the idea of running unit tests in parallel makes me very
 nervous. In general, across modules, I'd expect it to work, but there
 will be occasional cases where it will break.

Then you didn't write your unit-tests correctly. True unit tests-anyway. In fact, the very quality that makes unit tests so valuable (that they are independent of other code) is ruined by sharing state across tests. If you are going to share state, it really is one unit test.
 Across the unittest
 blocks in a single module, I'd be _very_ worried about breakage. There
 is nothing whatsoever in the language which guarantees that running
 them in parallel will work or even makes sense. All that protects us is
 the convention that unit tests are usually independent of each other,
 and in my experience, it's common enough that they're not independent
 that I think that blindly enabling parallelization of unit tests across
 a single module is definitely a bad idea.

I think that if we add the assumption, the resulting fallout would be easy to fix. Note that we can't require unit tests to be pure -- non-pure functions need testing too :) I can imagine that even if you could only parallelize 90% of unit tests, that would be an effective optimization for a large project. In such a case, the rare (and I mean rare to the point of I can't think of a single use-case) need to deny parallelization could be marked. -Steve
Apr 30 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 Apr 2014 11:43:31 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 Hello,


 A coworker mentioned the idea that unittests could be run in parallel  
 (using e.g. a thread pool). I've rigged things to run in parallel  
 unittests across modules, and that works well. However, this is too  
 coarse-grained - it would be great if each unittest could be pooled  
 across the thread pool. That's more difficult to implement.

I am not sure, but are unit-test blocks one function each, or one function per module? If the latter, that would have to be changed.
 This brings up the issue of naming unittests. It's becoming increasingly  
 obvious that anonymous unittests don't quite scale - coworkers are  
 increasingly talking about "the unittest at line 2035 is failing" and  
 such. With unittests executing in multiple threads and issuing e.g.  
 logging output, this is only likely to become more exacerbated. We've  
 resisted named unittests but I think there's enough evidence to make the  
 change.

I would note this enhancement, which Walter agreed should be done at DConf '13 ;) https://issues.dlang.org/show_bug.cgi?id=10023 Jacob Carlborg has tried to make this work, but the PR has not been pulled yet (I think it needs some updating at least, and there were some unresolved questions IIRC).
 Last but not least, virtually nobody I know runs unittests and then  
 main. This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 I think it's time to change that. We could do it the  
 non-backward-compatible way by redefining -unittest to instruct the  
 compiler to not run main. Or we could define another flag such as  
 -unittest-only and then deprecate the existing one.

The runtime can intercept this parameter. I would like a mechanism to run main decided at runtime. We need no compiler modifications to effect this.
 Thoughts? Would anyone want to work on such stuff?

I can probably take a look at changing the unittests to avoid main without a runtime parameter. I have a good grasp on how the pre-main runtime works, having rewritten the module constructor algorithm a while back. I am hesitant to run all unit tests in parallel without an opt-out mechanism. The above enhancement being implemented would give us some ways to play around, though. -Steve
Apr 30 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 30/04/2014 17:43, Andrei Alexandrescu a crit :
 Hello,


 A coworker mentioned the idea that unittests could be run in parallel
 (using e.g. a thread pool). I've rigged things to run in parallel
 unittests across modules, and that works well. However, this is too
 coarse-grained - it would be great if each unittest could be pooled
 across the thread pool. That's more difficult to implement.

Java, and when execution time grow TDD loose rapidly his efficiently. Some Eclipse's plug-ins are able to run them in parallel if I remember correctly.
 This brings up the issue of naming unittests. It's becoming increasingly
 obvious that anonymous unittests don't quite scale - coworkers are
 increasingly talking about "the unittest at line 2035 is failing" and
 such. With unittests executing in multiple threads and issuing e.g.
 logging output, this is only likely to become more exacerbated. We've
 resisted named unittests but I think there's enough evidence to make the
 change.

benchmarks,...). Unittests evolves with the rest of the code.
 Last but not least, virtually nobody I know runs unittests and then
 main. This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 I think it's time to change that. We could do it the
 non-backward-compatible way by redefining -unittest to instruct the
 compiler to not run main. Or we could define another flag such as
 -unittest-only and then deprecate the existing one.

 Thoughts? Would anyone want to work on such stuff?


 Andrei

Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 15:33:17 -0700
"H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> wrote:

 On Wed, Apr 30, 2014 at 02:48:38PM -0700, Jonathan M Davis via
 Digitalmars-d wrote:
 On Wed, 30 Apr 2014 21:09:14 +0100
 Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 In which case D is wrong to allow them in the unittest blocks and
 should introduce a new way of handling these tests. And even then
 all tests can and should be parallelized. If they cannot be then
 there is an inappropriate dependency.

Why? Because Andrei suddenly proposed that we parallelize unittest blocks? If I want to test a function, I'm going to put a unittest block after it to test it. If that means accessing I/O, then it means accessing I/O. If that means messing with mutable, global variables, then that means messing with mutable, global variables. Why should I have to put the tests elsewhere or make is that they don't run whenthe -unttest flag is used just because they don't fall under your definition of "unit" test?

What about allowing pure marking on unittests, and those unittests that are marked pure will be parallelized, and those that aren't marked will be run serially?

I think that that would work, and if we added purity inferrence to unittest blocks as Nordlow suggests, then you wouldn't even have to mark them as pure unless you wanted to enforce that it be runnable in parallel. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 20:33:06 -0400
Steven Schveighoffer via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On Wed, 30 Apr 2014 13:50:10 -0400, Jonathan M Davis via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 
 On Wed, 30 Apr 2014 08:59:42 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

In general, I agree. In reality, there are times when having state across unit tests makes sense - especially when there's expensive setup required for the tests.

int a; unittest { // set up a; } unittest { // use a; } ==> unittest { int a; { // set up a; } { // use a; } } It makes no sense to do it the first way, you are not gaining anything.

It can make sense to do it the first way when it's more like LargeDocumentOrDatabase foo; unittest { // set up foo; } unittest { // test something using foo } unittest { // do other tests using foo which then take advantage of changes made // by the previous test rather than doing all of those changes to // foo in order to set up this test } In general, I agree that tests shouldn't be done that way, and I don't think that I've ever done it personally, but I've seen it done, and for stuff that requires a fair bit of initialization, it can save time to have each test build on the state of the last. But even if we all agree that that sort of testing is a horrible idea, the language supports it right now, and automatically parallelizing unit tests will break any code that does that.
 Honestly, the idea of running unit tests in parallel makes me very
 nervous. In general, across modules, I'd expect it to work, but
 there will be occasional cases where it will break.

Then you didn't write your unit-tests correctly. True unit tests-anyway. In fact, the very quality that makes unit tests so valuable (that they are independent of other code) is ruined by sharing state across tests. If you are going to share state, it really is one unit test.

All it takes is that tests in two separate modules which have separate functionality access the file system or sockets or some other system resource, and they could end up breaking due to the fact that the other test is messing with the same resource. I'd expect that to be a relatively rare case, but it _can_ happen, so simply parallelizing tests across modules does risk test failures that would not have occurred otherwise.
 Across the unittest
 blocks in a single module, I'd be _very_ worried about breakage.
 There is nothing whatsoever in the language which guarantees that
 running them in parallel will work or even makes sense. All that
 protects us is the convention that unit tests are usually
 independent of each other, and in my experience, it's common enough
 that they're not independent that I think that blindly enabling
 parallelization of unit tests across a single module is definitely
 a bad idea.

I think that if we add the assumption, the resulting fallout would be easy to fix. Note that we can't require unit tests to be pure -- non-pure functions need testing too :)

Sure, they need testing. Just don't test them in parallel, because they're not guaranteed to work in parallel. That guarantee _does_ hold for pure functions, because they don't access global, mutable state. So, we can safely parallelize a unittest block that is pure, but we _can't_ safely paralellize one that isn't - not in a guaranteed way.
 I can imagine that even if you could only parallelize 90% of unit
 tests, that would be an effective optimization for a large project.
 In such a case, the rare (and I mean rare to the point of I can't
 think of a single use-case) need to deny parallelization could be
 marked.

std.file's unit tests would break immediately. It wouldn't surprise me if std.socket's unit tests broke. std.datetime's unit tests would probably break on Posix systems, because some of them temporarily set the local time zone - which sets it for the whole program, not just the current thread (those tests aren't done on Windows, because Windows only lets you set it for the whole OS, not just the program). Any tests which aren't pure risk breakage due to changes in whatever global, mutable state they're accessing. I would strongly argue that automatically parallelizing any unittest block which isn't pure is a bad idea, because it's not guaranteed to work, and it _will_ result in bugs in at least some cases. If we make it so that unittest blocks have their purity inferred (and allow you to mark them as pure to enforce that they be pure if you want to require it), then any unittest blocks which can safely be parallelized will be known, and the test runner could then parallelize those unittest functions and then _not_ parallelize the ones that it can't guarantee are going to work in parallel. So, then we get safe, unittest parallelization without having to insist that folks write their unit tests in a particular way or that they do or don't do particular things in a unit test. And maybe we can add add some sort of UDA to tell the test runner that an impure test can be safely parallelized, but automatically parallelizing impure unittest functions would be akin to automatically treating system functions as safe just because we thought that only safe code should be used in this particular context. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 14:35:45 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:
 Agreed. I think we should look into parallelizing all unittests. --

I'm all for parallelizing all unittest blocks that are pure, as doing so would be safe, but I think that we're making a big mistake if we try and insist that all unittest blocks be able to be run in parallel. Any that aren't pure are not guaranteed to be parallelizable, and any which access system resources or other global, mutable state stand a good chance of breaking. If we make it so that the functions generated from unittest blocks have their purity inferred, then any unittest block which can safely be parallelized could then be parallelized by the test runner based on their purity, and any impure unittest functions could then be safely run in serial. And if you want to make sure that a unittest block is parallizable, then you can just explicitly mark it as pure. With that approach, we don't risk breaking existing unit tests, and it allows tests that need to not be run in parallel to work properly by guaranteeing that they're still run serially. And it even make it so that many tests are automatically parallelizable without the programmer having to do anything special for it. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Wednesday, 30 April 2014 at 15:43:35 UTC, Andrei Alexandrescu 
wrote:
 This brings up the issue of naming unittests. It's becoming 
 increasingly obvious that anonymous unittests don't quite scale

A message structured like this would be awesome. Unittest Failed foo.d:345 Providing null input throws exception
 Last but not least, virtually nobody I know runs unittests and 
 then main. This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
    ...
 }

 I think it's time to change that. We could do it the 
 non-backward-compatible way by redefining -unittest to instruct 
 the compiler to not run main. Or we could define another flag 
 such as -unittest-only and then deprecate the existing one.

I would like to see -unittest redefined.
Apr 30 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 22:32:33 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On 4/30/14, 10:01 PM, Jonathan M Davis via Digitalmars-d wrote:
 I'm all for parallelizing all unittest blocks that are pure, as
 doing so would be safe, but I think that we're making a big mistake
 if we try and insist that all unittest blocks be able to be run in
 parallel. Any that aren't pure are not guaranteed to be
 parallelizable, and any which access system resources or other
 global, mutable state stand a good chance of breaking.

There are a number of assumptions here: (a) most unittests that can be effectively parallelized can be actually inferred (or declared) as pure; (b) most unittests that cannot be inferred as pure are likely to break; (c) it's a big deal if unittests break. I question all of these assumptions. In particular I consider unittests that depend on one another an effective antipattern that needs to be eradicated.

Even if they don't depend on each other, they can depend on the system. std.file's unit tests will break if we parallelize them, because it operates on files and directories, and many of those tests operate on the same temp directories. That can be fixed by changing the tests, but it will break the tests. Other tests _can't_ be fixed if we force them to run in parallel. For instance, some of std.datetime's unit tests set the local time zone of the system in order to test that LocalTime works correctly. That sets it for the whole program, so all threads will be affected even if they're running other tests. Right now, this isn't a problem, because those tests set the timezone at their start and reset it at their end. But if they were made to run in parallel with any other tests involving LocalTime, there's a good chance that those tests would have random test failures. They simply can't be run in parallel due to a system resource that we can't make thread-local. So, regardless of how we want to mark up unittest blocks as parallelizable or not parallelizable (be it explicit, implict, using pure, or using something else), we do need a way to make it so that a unittest block is not run in parallel with any other unittest block. We can guarantee that pure functions can safely be run in parallel. We _cannot_ guarantee that impure functions can safely be run in parallel. I'm sure that many impure unittest functions could be safely run in parallel, but it would require that the programmer verify that if we don't want undefined behavior - just like programmers have to verify that system code is actually safe. Simply running all unittest blocks in parallel is akin to considering system code safe in a particular piece of code simply because by convention that code should be safe. pure allows us to detect guaranteed, safe parallelizability. If we want to define some other way to make it so a unittest block can be marked as parallelizable regardless of purity, then fine. But automatically parallelizing impure functions means that we're going to have undefined behavior for those unittest functions, and I really think that that is a bad idea - in addition to the fact that some unittest blocks legitimately cannot be run in parallel due to the use of system resources, so parallelizing them _will_ not only break them but make them impossible to write in a way that's not broken without adding mutexes to the unittest blocks to stop the test runner from running them in parallel. And IMHO, if we end up having to do that anywhere, we've done something very wrong with how unit tests work. - Jonathan M Davis
Apr 30 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 30 April 2014 at 20:20:26 UTC, Jacob Carlborg wrote:
 On 2014-04-30 19:30, Dicebot wrote:
 I believe only missing step right now is propagation of UDA's 
 to RTInfo
 when demanded. Everything else can be done as Phobos solution.

I don't see why this is necessary for this case.

It is not strictly necessary but you can't reliably get all unit test blocks during compile-time (must be transitively imported) and current run-time reflection for tests is missing any data but actual function pointers. I am personally perfectly satisfied with "single root module imports all" approach but it is likely to be complained if proposed as "standard" way.
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 30 April 2014 at 21:49:06 UTC, Jonathan M Davis via 
Digitalmars-d wrote:
 On Wed, 30 Apr 2014 21:09:14 +0100
 Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> 
 wrote:

 On Wed, 2014-04-30 at 11:19 -0700, Jonathan M Davis via 
 Digitalmars-d
 wrote:
 unittest blocks just like any other unit test. I would very 
 much
 consider std.file's tests to be unit tests. But even if you 
 don't
 want to call them unit tests, because they access the file 
 system,
 the reality of the matter is that tests like them are going 
 to be
 run in unittest blocks, and we have to take that into 
 account when
 we decide how we want unittest blocks to be run (e.g. whether
 they're parallelizable or not).

In which case D is wrong to allow them in the unittest blocks and should introduce a new way of handling these tests. And even then all tests can and should be parallelized. If they cannot be then there is an inappropriate dependency.

Why? Because Andrei suddenly proposed that we parallelize unittest blocks? If I want to test a function, I'm going to put a unittest block after it to test it. If that means accessing I/O, then it means accessing I/O. If that means messing with mutable, global variables, then that means messing with mutable, global variables. Why should I have to put the tests elsewhere or make is that they don't run whenthe -unttest flag is used just because they don't fall under your definition of "unit" test?

You do this because unit tests must be fast. You do this because unit tests must be naively parallel. You do this because unit tests verify basic application / library sanity and expected to be quickly run after every build in deterministic way (contrary to full test suite which can take hours). Also you do that because doing _reliably_ correct tests with I/O is relatively complicated and one does not want to pollute actual source modules with all environment checks. In the end it is all about supporting quick edit-compile-test development cycle.
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 30 April 2014 at 21:09:51 UTC, Átila Neves wrote:
 I don't know about anyone else, but I make my tests fail a 
 lot.

I think this is key difference. For me failing unit test is always exceptional situation.

I TDD a lot. Tests failing are normal. Not only that, I refactor a lot as well. Which causes tests to fail. Fortunately I have tests failing to tell me I screwed up.

I dream of a day when TDD crap will be finally discarded and fade into oblivion.
 Even if failing tests were exceptional, I still want everything 
 I just mentioned.

Probably. But will you still _need_ it? ;)
And if test group is complex
 enough to require categorization then either my code is not 
 procedural enough or module is just too big and needs to be 
 split.

And when I split them I put them into a subcategory.

This is somewhat redundant as they are already categorized by module / package.
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 01:45:21 UTC, Xavier Bigand wrote:
 Splitting all features at an absolute atomic level can be 
 achieve for open-source libraries, but it's pretty much 
 impossible for an industrial software. Why being so restrictive 
 when it's possible to support both vision by extending a little 
 the language by something already logical?

You are pretty much saying here "writing good code is possible for open-source libraries but not for industrial software".
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 23:56:53 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:
 I don't think undefined behavior is at stake here, and I find the
 simile invalid. Thread isolation is a done deal in D and we may as
 well take advantage of it. Worse that could happen is that a unittest
 sets a global and surprisingly the next one doesn't "see" it.
 
 At any rate I think it's pointless to insist on limiting parallel 
 running to pure - let me just say I understood the point (thanks) so 
 there is no need to restate it, and that I think it doesn't take us a 
 good place.

I'm only arguing for using pure on the grounds that it _guarantees_ that the unittest block is safely parallelizable. If we decide that that guarantee isn't necessary, then we decide that it isn't necessary, though I definitely worry that not having that guarantee will be problematic. I do agree though that D's thread-local by default helps quite a bit in ensuring that most tests will be runnable in parallel. However, if we went with purity to indicate parallelizability, I could easily see doing it implicitly based on purity and allowing for a UDA or somesuch which marked a unittest block as "trusted pure" so that it could be run in parallel. So, I don't think that going with pure would necessarily be too restrictive. It just would require that the programmer do some extra work to be able to treat a unittest block as safely parallelizable when the compiler couldn't guarantee that it was. Ultimately, my biggest concern here is that it be possible to guarantee that a unittest block is not run in parallel with any other unittest block if that particular unittest requires it for any reason, and some folks seem to be arguing that such tests are always invalid, and I want to make sure that we don't ever consider that to be the case for unittest blocks in D. If we do parallel by default and allow for some kind of markup to make a unittest block serial, then that can work. I fully expect that switching to parallel by default would break a number of tests, which I do think is a problem (particularly since a number of those tests will be completely valid), but it could also be an acceptable one - especially if for the most part, the code that it breaks is badly written code. Regardless, we will need to make sure that we message the change clearly in order to ensure that a minimal number of people end up with random test failures due to the change. On a side note, regardless of whether we want to use purity to infer paralellizability, I think that it's very cool that we have the capability to do so if we so choose, whereas most other languages have no way of even coming close to being able to tell whether a function can be safely parallelized or not. The combination of attributes such as pure and compile-time inference is very cool indeed. - Jonathan M Davis
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 04:50:30 UTC, Jonathan M Davis via 
Digitalmars-d wrote:
 std.file's unit tests would break immediately. It wouldn't 
 surprise me
 if std.socket's unit tests broke. std.datetime's unit tests 
 would
 probably break on Posix systems, because some of them 
 temporarily set
 the local time zone - which sets it for the whole program, not 
 just the
 current thread (those tests aren't done on Windows, because 
 Windows only
 lets you set it for the whole OS, not just the program). Any 
 tests which
 aren't pure risk breakage due to changes in whatever global, 
 mutable
 state they're accessing.

We really should think about separating Phobos tests into unit tests and higher level ones (in separate top-level source folder). The fact that importing std.file in my code with `rdmd -unittest` may trigger file I/O makes me _extremely_ disgusted. How did we even get here? >_<
May 01 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 On 4/30/2014 8:54 AM, bearophile wrote:
 I'd also like some built-in way (or partially built-in) to use 
 a module only as
 "main module" (to run its demos) or as module to be imported. 
 This problem is
 solved in Python with the "if __name__ == "__main__":" idiom.

dmd foo.d -unittest -main

I think you are greatly missing the point. The unittests are mainly for the developer, while the demo is for the user of the demo. The unittests validate the code, while the demo shows how to use the module (and surely the demo is not meant to run in parallel with the other unittests). Sometimes the demo part is more than a demo, it contains usable and useful code, with a command-line interface, to allow stand alone usage of the module functionality. Currently this is how I do it: module foobar; // Here functions and // classes with their unittests version (foobar_main) { void main() { // foobar module demo code here. } } I can't use just "version(main)", because the module could import other modules with their own demo sections. So I need something to tell them apart from each other. This could be simplified with a solution similar to the Python one: version (__is_main) { void main() { // foobar module demo code here. } } Now if I have module A and B, where B imports A, and both have the demo main, I can use this to compile A stand-alone with its demo: dmd A.d And I can use this to not compile the demo of A and compile the demo section of B: dmd A.d B.d This is just the basic idea, and perhaps people suggested something better than this. Bye, bearophile
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 01 May 2014 07:26:59 +0000
Dicebot via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Thursday, 1 May 2014 at 04:50:30 UTC, Jonathan M Davis via 
 Digitalmars-d wrote:
 std.file's unit tests would break immediately. It wouldn't 
 surprise me
 if std.socket's unit tests broke. std.datetime's unit tests 
 would
 probably break on Posix systems, because some of them 
 temporarily set
 the local time zone - which sets it for the whole program, not 
 just the
 current thread (those tests aren't done on Windows, because 
 Windows only
 lets you set it for the whole OS, not just the program). Any 
 tests which
 aren't pure risk breakage due to changes in whatever global, 
 mutable
 state they're accessing.

We really should think about separating Phobos tests into unit tests and higher level ones (in separate top-level source folder). The fact that importing std.file in my code with `rdmd -unittest` may trigger file I/O makes me _extremely_ disgusted. How did we even get here? >_<

Honestly, I see no problem with std.file's unit tests triggering I/O. That's what the module _does_. And it specifically uses the system's temp directory so that it doesn't screw with anything else on the system. Separating the tests out into some other set of tests wouldn't buy us anything IMHO. The tests need to be run regardless, and they need to be run with the same frequency regardless. Splitting those tests out would just make them harder for developers to run, because now they'd have to worry about running two sets of tests instead of just one. As far as I can see, splitting out tests that do I/O would be purely for ideological reasons and would be of no practical benefit. In fact, it would be _less_ practical if we were to do so. - Jonathan M Davis
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 07:47:27 UTC, Jonathan M Davis via 
Digitalmars-d wrote:
 Honestly, I see no problem with std.file's unit tests 
 triggering I/O.
 That's what the module _does_. And it specifically uses the 
 system's
 temp directory so that it doesn't screw with anything else on 
 the
 system. Separating the tests out into some other set of tests 
 wouldn't
 buy us anything IMHO. The tests need to be run regardless, and 
 they
 need to be run with the same frequency regardless. Splitting 
 those
 tests out would just make them harder for developers to run, 
 because
 now they'd have to worry about running two sets of tests 
 instead of
 just one. As far as I can see, splitting out tests that do I/O 
 would be
 purely for ideological reasons and would be of no practical 
 benefit. In
 fact, it would be _less_ practical if we were to do so.

 - Jonathan M Davis

I have just recently went through some of out internal projects removing all accidental I/O tests for the very reason that /tmp was full and those tests were randomly failing when testing my own program that have used the library. This _sucks_. You can't do any test with I/O without verifying the environment (free space, concurrent access from other processes, file system access etc). And once you do it properly such beast has no longer fits into the same module because of sheer size. There is a very practical reason to separate tests - to become sure that you always can run -unittest build to verify basic sanity of your program and it will never spuriously fail or take eternity to complete.
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 07:32:43 UTC, bearophile wrote:
 This is just the basic idea, and perhaps people suggested 
 something better than this.

 Bye,
 bearophile

Yeah I sometimes have commented out main() for that purpose. Sounds like a useful generic addition.
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 30 Apr 2014 13:31:30 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 I'd argue for regular identifiers instead of strings - they can be
 seen in stack traces, accessed with __FUNCTION__ etc. -- Andrei

If we actually want to make unittests work just like functions (__FUNCTION__, identifier which are visible in stacktraces) then we could also simply declare functions and mark them as ( )unittest: unittest void myUnittest() { } This then allows for further improvements in the future, for example a unit test can then return a result value (skipped, error, ..) or it could optionally receive parameters (like some kind of state from a unittest framework)
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 30 Apr 2014 22:19:24 +0200
schrieb Jacob Carlborg <doob me.com>:

 __traits(getUnitTests) 

I've always wondered, but never asked: Doesn't __traits(getUnitTests) usage suffer from the same problem std.benchmark had? That in order to make it work for all modules you a) have to explicitly mention every module b) use module constructors, leading to module constructor dependency hell (afaik the main reason we don't have a std.benchmark now) ?
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 01 May 2014 09:38:51 +0000
schrieb "Dicebot" <public dicebot.lv>:

 On Thursday, 1 May 2014 at 08:51:47 UTC, Johannes Pfau wrote:
 
 You only need to make sure all modules are transitively imported
 from initial one. Everything else can be done via recursive
 reflection with __traits(allMembers), module constructors are not
 needed either.
 
 One can also generate special test entry module that imports all
 project modules using build system help.

Is there some boost licensed reflection code somewhere which shows how to do recursive reflection correctly? I know the basic idea but stuff like dealing with protection of members would take some time to figure that out. Maybe someone familiar with the details could sketch up a quick std.test proof-of-concept which parses the following (with all necessary protection checks + detecting unittest blocks in classes/structs): ----------- unittest { } _unittest void namedTest() { } ---------- Andrei do you think having to explicitly import modules to be tested is an issue? Apart from this limitation such an approach sounds great to me. There are quite some possibilities how user frameworks could extend std.test so it sounds like an interesting idea. (And std.benchmark could work in the same way)
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 30 Apr 2014 09:02:54 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 To summarize: It provides a function pointer for every  unit test to
 druntime or user code. This is actually easy to do. Naming tests
 requires changes in the parser, but I guess that shouldn't be
 difficult either.  

That's fantastic, would you be willing to reconsider that work?

Are you still interested in this? I guess we could build a std.test phobos module to completely replace the current unittest implementation, see: http://forum.dlang.org/post/ljtbch$lg6$1 digitalmars.com This seems to be a better solution, especially considering extension possibilities.
May 01 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all 
 unittests. -- Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 01 May 2014 13:24:07 +0200
schrieb Jacob Carlborg <doob me.com>:

 On 2014-05-01 11:37, Dicebot wrote:
 
 You only need to make sure all modules are transitively imported
 from initial one.

The solution for that would be RMInfo [1], like RTInfo but for modules instead of types. [1] https://github.com/D-Programming-Language/dmd/pull/2271

But on a quick look I don't understand how this (or DRT #775) completely solves the issue. Now you don't have to import the modules anymore, which is a step forward, but let's say I want to used this to find all SubClasses of a class. Now I can inspect that code in CTFE, but how do I build a list of all subclasses? In the most generic way (with DLLs/shared libraries) this can only work if code can be executed at runtime, so we'd need a way to emit module constructors AFAICS. These should be able to avoid the usual constructor dependency rules though.
May 01 2014
prev sibling next sibling parent Byron <byron.heads gmail.com> writes:
On Thu, 01 May 2014 11:44:11 +0000, w0rp wrote:

 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

Running tests in random order helps find hidden dependencies, but I wouldn't want it as a default. I lot of unittesting libraries offer this. If you don't run tests often it doesn't help much, but if you do TDD it can help.
May 01 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
 I dream of a day when TDD crap will be finally discarded and 
 fade into oblivion.

I think most people who don't like TDD don't fully understand it. At the same time I think people who like TDD tend to abuse it. Either way, I like it, do it, and want my workflow to be the best it is with it.
 Even if failing tests were exceptional, I still want 
 everything I just mentioned.

Probably. But will you still _need_ it? ;)

Yes. :P
And if test group is complex
 enough to require categorization then either my code is not 
 procedural enough or module is just too big and needs to be 
 split.

And when I split them I put them into a subcategory.

This is somewhat redundant as they are already categorized by module / package.

Which is why my library uses the fully qualified name of the test to categorise them. Atila
May 01 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all 
 unittests. -- Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that. Atila
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 09:26:39 -0400, Byron <byron.heads gmail.com> wrote:

 On Thu, 01 May 2014 11:44:11 +0000, w0rp wrote:

 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

Running tests in random order helps find hidden dependencies, but I wouldn't want it as a default. I lot of unittesting libraries offer this. If you don't run tests often it doesn't help much, but if you do TDD it can help.

Note the order of unit tests is defined by druntime. It can easily be modified. -Steve
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 00:49:53 -0400, Jonathan M Davis via Digitalmars-d  
<digitalmars-d puremagic.com> wrote:

 On Wed, 30 Apr 2014 20:33:06 -0400
 Steven Schveighoffer via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 On Wed, 30 Apr 2014 13:50:10 -0400, Jonathan M Davis via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wed, 30 Apr 2014 08:59:42 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 On 4/30/14, 8:54 AM, bearophile wrote:
 Andrei Alexandrescu:

 A coworker mentioned the idea that unittests could be run in
 parallel

In D we have strong purity to make more safe to run code in parallel: pure unittest {}

This doesn't follow. All unittests should be executable concurrently. -- Andrei

In general, I agree. In reality, there are times when having state across unit tests makes sense - especially when there's expensive setup required for the tests.

int a; unittest { // set up a; } unittest { // use a; } ==> unittest { int a; { // set up a; } { // use a; } } It makes no sense to do it the first way, you are not gaining anything.

It can make sense to do it the first way when it's more like LargeDocumentOrDatabase foo; unittest { // set up foo; } unittest { // test something using foo } unittest { // do other tests using foo which then take advantage of changes made // by the previous test rather than doing all of those changes to // foo in order to set up this test } In general, I agree that tests shouldn't be done that way, and I don't think that I've ever done it personally, but I've seen it done, and for stuff that requires a fair bit of initialization, it can save time to have each test build on the state of the last. But even if we all agree that that sort of testing is a horrible idea, the language supports it right now, and automatically parallelizing unit tests will break any code that does that.

I recommend optimizing using a function. i.e.: version(unittest) { LargeDocumentOrDatabase foo; auto getFoo() {/* check to see if foo is set up, return it*/} } I understand what you are saying. I think the largest problem with parallelizing unit tests is that people haven't been careful to make sure that's possible. Now they should, or face the consequences. The point I was making, however, is that within a module, you can choose whether you want parallel or serial unit tests. If you want parallel, separate them into multiple unittest blocks. If you want serial, put them in one. For the super-rare cases where it needs to be serial, put them in one. It's not hard.
 Honestly, the idea of running unit tests in parallel makes me very
 nervous. In general, across modules, I'd expect it to work, but
 there will be occasional cases where it will break.

Then you didn't write your unit-tests correctly. True unit tests-anyway. In fact, the very quality that makes unit tests so valuable (that they are independent of other code) is ruined by sharing state across tests. If you are going to share state, it really is one unit test.

All it takes is that tests in two separate modules which have separate functionality access the file system or sockets or some other system resource, and they could end up breaking due to the fact that the other test is messing with the same resource. I'd expect that to be a relatively rare case, but it _can_ happen, so simply parallelizing tests across modules does risk test failures that would not have occurred otherwise.

Right, and with the knowledge that unit tests are being run in parallel, one can trivially change their design to fix the problem. I agree my assumptions were not what you were thinking of. I wasn't thinking of shared system resources. But this isn't too difficult to figure out. I do think there should be a way to mark a unit test as "don't parallelize this".
 Across the unittest
 blocks in a single module, I'd be _very_ worried about breakage.
 There is nothing whatsoever in the language which guarantees that
 running them in parallel will work or even makes sense. All that
 protects us is the convention that unit tests are usually
 independent of each other, and in my experience, it's common enough
 that they're not independent that I think that blindly enabling
 parallelization of unit tests across a single module is definitely
 a bad idea.

I think that if we add the assumption, the resulting fallout would be easy to fix. Note that we can't require unit tests to be pure -- non-pure functions need testing too :)

Sure, they need testing. Just don't test them in parallel, because they're not guaranteed to work in parallel. That guarantee _does_ hold for pure functions, because they don't access global, mutable state. So, we can safely parallelize a unittest block that is pure, but we _can't_ safely paralellize one that isn't - not in a guaranteed way.

A function may be impure, but run in a pure way.
 I can imagine that even if you could only parallelize 90% of unit
 tests, that would be an effective optimization for a large project.
 In such a case, the rare (and I mean rare to the point of I can't
 think of a single use-case) need to deny parallelization could be
 marked.

std.file's unit tests would break immediately. It wouldn't surprise me if std.socket's unit tests broke. std.datetime's unit tests would probably break on Posix systems, because some of them temporarily set the local time zone - which sets it for the whole program, not just the current thread (those tests aren't done on Windows, because Windows only lets you set it for the whole OS, not just the program). Any tests which aren't pure risk breakage due to changes in whatever global, mutable state they're accessing.

It depends on what the impure function is doing. Anything that's using the same temporary file should not be, that's easy. Anything that's using the same socket port should not be, that's easy. Anything that requires using the local time zone should be done in a single unit test. Most everything in std.datetime should use a defined time zone instead of local time. Unit tests should be as decoupled from system/global state as possible.
 I would strongly argue that automatically parallelizing any unittest
 block which isn't pure is a bad idea, because it's not guaranteed to
 work, and it _will_ result in bugs in at least some cases. If we make it
 so that unittest blocks have their purity inferred (and allow you to
 mark them as pure to enforce that they be pure if you want to require
 it), then any unittest blocks which can safely be parallelized will be
 known, and the test runner could then parallelize those unittest
 functions and then _not_ parallelize the ones that it can't guarantee
 are going to work in parallel.

It would be at least a start, and I agree it would be an easy way to add some level of parallelism without breaking any existing tests. But I fear that there are many things that would be excluded incorrectly. Take for example, std.datetime. The constructor for SysTime has this line in it: _timezone = tz is null ? LocalTime() : tz; All unit tests that pass in a specific tz (such as UTC) could be pure calls. But because of that line, they can't be!
 So, then we get safe, unittest parallelization without having to insist
 that folks write their unit tests in a particular way or that they do or
 don't do particular things in a unit test. And maybe we can add add
 some sort of UDA to tell the test runner that an impure test can be
 safely parallelized, but automatically parallelizing impure unittest
 functions would be akin to automatically treating  system functions as
  safe just because we thought that only  safe code should be used in
 this particular context.

As I pointed out to Andrei, there is an enhancement, with a pull request even, that would potentially add the ability to detect UDAs on unit tests, giving us the ability to name them or to mark them however we want, so the runtime can take appropriate action. https://issues.dlang.org/show_bug.cgi?id=10023 -Steve
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu 
wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal 
 projects removing
 all accidental I/O tests for the very reason that /tmp was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics? Including automated test servers? Really? Bunch of stuff can go wrong. Higher level tests verify their expectations from environment (if written carefully), which is impractical to do in unit tests. Also such reliance on environment makes running in parallel impossible without explicit resource dependency tracking for any kind of bigger test suite.
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 12:04:57 UTC, Xavier Bigand wrote:
 It's just a lot harder when you are under pressure.
 I am working for a very small company and our dead lines 
 clearly doesn't help us with that, and because I am in the 
 video game industry it's not really critical to have small bugs.

 Not every body have the capacity or resources (essentially 
 time) to design his code in the pure conformance of unittests 
 definition, and IMO isn't not an excuse to avoid tests 
 completely.
 If a language/standard library can help democratization of 
 tests it's a good thing, so maybe writing tests have to stay 
 relatively simple and straightforward.

 My point is just when you are doing things only for you it's 
 often simpler to them like they must be.

I know that and don't have luxury of time for perfect tests either :) But it is more about state of mind than actual time consumption - once you start keeping higher level tests with I/O separate and making observation how some piece of functionality can be tested in contained way, you approach to designing modules changes. At some point one simply starts to write unit test friendly modules from the very first go, it is all about actually thinking into it. Using less OOP and more functional programming helps with that btw :) I can readily admit that in real industry projects one is likely to do many different "dirty" things and this is inevitable. What I do object to is statement that this is the way to go in general, especially in language standard library.
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 11:04:31 -0400, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/1/14, 4:05 AM, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --
 Andrei

I recommend running the tests in random order as well.

Great idea! -- Andrei

I think we can configure this at runtime. Imagine, you have multiple failing unit tests. You see the first failure. You find the issue, try and fix the problem, or instrument it, and now a DIFFERENT test fails. Now focus on that one, yet a different one fails. This is just going to equal frustration. If you want to run random, we can do that. If you want to run in order, that also should be possible. In fact, while debugging, you need to run them in order, and serially. -Steve
May 01 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu 
wrote:
 On 5/1/14, 8:04 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu 
 wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal 
 projects removing
 all accidental I/O tests for the very reason that /tmp was 
 full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics?

I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei

It got full because of tests (surprise!). Your actions?
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 12:07:19 -0400, Dicebot <public dicebot.lv> wrote:

 On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 8:04 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal projects  
 removing
 all accidental I/O tests for the very reason that /tmp was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics?

I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei

It got full because of tests (surprise!). Your actions?

It would be nice to have a uniform mechanism to get a unique system-dependent file location for each specific unit test. The file should automatically delete itself at the end of the test. -Steve
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 10:01:19 -0400, Atila Neves <atila.neves gmail.com>  
wrote:

 On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all unittests. --  
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that.

But not a great way to debug it. If your test failure depends on ordering, then the next run will be random too. Proposal runtime parameter for pre-main consumption: ./myprog --rndunit[=seed] To run unit tests randomly. Prints out as first order of business the seed value before starting. That way, you can repeat the exact same ordering for debugging. -Steve
May 01 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 1 May 2014 at 17:24:58 UTC, Andrei Alexandrescu 
wrote:
 On 5/1/14, 10:09 AM, Steven Schveighoffer wrote:
 On Thu, 01 May 2014 12:07:19 -0400, Dicebot 
 <public dicebot.lv> wrote:

 On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu 
 wrote:
 On 5/1/14, 8:04 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei 
 Alexandrescu wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal 
 projects
 removing
 all accidental I/O tests for the very reason that /tmp 
 was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics?

I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei

It got full because of tests (surprise!). Your actions?

It would be nice to have a uniform mechanism to get a unique system-dependent file location for each specific unit test. The file should automatically delete itself at the end of the test.

Looks like /tmp (%TEMP% or C:\TEMP in Windows) in conjunction with the likes of mkstemp is what you're looking for :o). Andrei

It hasn't been C:\TEMP for almost 13 years (before Windows XP which is now also end-of-life). Use GetTempPath. http://msdn.microsoft.com/en-us/library/windows/desktop/aa364992(v=vs.85).aspx
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 13:25:00 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 5/1/14, 10:09 AM, Steven Schveighoffer wrote:
 On Thu, 01 May 2014 12:07:19 -0400, Dicebot <public dicebot.lv> wrote:

 On Thursday, 1 May 2014 at 15:37:21 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 8:04 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 14:55:50 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 1:34 AM, Dicebot wrote:
 I have just recently went through some of out internal projects
 removing
 all accidental I/O tests for the very reason that /tmp was full

Well a bunch of stuff will not work on a full /tmp. Sorry, hard to elicit empathy with a full /tmp :o). -- Andrei

So you are OK with your unit tests failing randomly with no clear diagnostics?

I'm OK with my unit tests failing on a machine with a full /tmp. The machine needs fixing. -- Andrei

It got full because of tests (surprise!). Your actions?

It would be nice to have a uniform mechanism to get a unique system-dependent file location for each specific unit test. The file should automatically delete itself at the end of the test.

Looks like /tmp (%TEMP% or C:\TEMP in Windows) in conjunction with the likes of mkstemp is what you're looking for :o).

No, I'm looking for unittest_getTempFile(Line = __LINE__, File = __FILE__)(), which handles all the magic of opening a temporary file, allowing me to use it for the unit test, and then closing and deleting it at the end, when the test passes. -Steve
May 01 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Thursday, 1 May 2014 at 17:04:53 UTC, Xavier Bigand wrote:
 Le 01/05/2014 16:01, Atila Neves a écrit :
 On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all 
 unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that. Atila


Tests shouldn't be run in a random order all of the time, perhaps once in a while, manually. Having continuous integration randomly report build failures is crap. Either you should always see a build failure, or you shouldn't see it. You can only test things which are deterministic, at least as far as what you observe. Running tests in a random order should be something you do manually, only when you have some ability to figure out why the tests just failed.
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 01 May 2014 10:42:54 -0400
Steven Schveighoffer via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On Thu, 01 May 2014 00:49:53 -0400, Jonathan M Davis via
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 
 On Wed, 30 Apr 2014 20:33:06 -0400
 Steven Schveighoffer via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:


 I do think there should be a way to mark a unit test as "don't
 parallelize this".

Regardless what our exact solution is, a key thing is that we need to be able have both tests which are run in parallel and tests which are run in serial. Switching to parallel by default will break code, but that may be acceptable. And I'm somewhat concerned about automatically parallelizing unit tests which aren't pure just because it's still trivial to write unittest blocks that aren't safely parallelizable (even if most such examples typically aren't good practice) whereas they'd work just fine now. But ultimately, my main concern is that we not enforce that all unit tests be parallelized, because that precludes certain types of tests.
 A function may be impure, but run in a pure way.

True. The idea behind using purity is that it guarantees that the unittest blocks would be safely parallelizable. But even if we were to go with purity, that doesn't preclude having some way to mark a unittest as parallelizable in spite of its lack of purity. It just wouldn't be automatic.
 Anything that requires using the local time zone should be done in a
 single unit test. Most everything in std.datetime should use a
 defined time zone instead of local time.

Because LocalTime is the default timezone, most of the tests use it. In general, I think that that's fine and desirable, because LocalTime is what most everyone is going to be using. Where I think that it actually ends up being a problem (and will eventually necessitate that I rewrite a number of the tests - possibly most of them) is when tests end up making assumptions that can break in certain time zones. So, in the long run, I expect that far fewer tests will use LocalTime than is currently the case, but I don't think that I agree that it should be avoided on quite the level that you seem to. It is on my todo list though to go over std.datetime's unit tests and make it stop using LocalTime where that will result in the tests failing in some time zones.
 Take for example, std.datetime. The constructor for SysTime has this
 line in it:
 
 _timezone = tz is null ? LocalTime() : tz;
 
 All unit tests that pass in a specific tz (such as UTC) could be
 pure calls. But because of that line, they can't be!

Pretty much nothing involving SysTime is pure, because adjTime can't be pure, because LocalTime's conversion functions can't be pure, because it calls the system's functions to do the conversions. So, very few of SysTime's unit tests could be parallelized based on purity. The constructor is just one of many places where SysTime can't be pure. So, it's an example of the types of tests that would have to be marked as explicitly parallelizable if we used purity as a means of determining automatic parallelizabitily. - Jonathan M Davis
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 01 May 2014 08:07:51 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 5/1/14, 4:31 AM, Johannes Pfau wrote:
  Andrei do you think having to explicitly import modules to be
 tested is an issue?

Well it kinda is. All that's written on the package is "unittest", we should add no fine print to it. -- Andrei

It'd be possible to make it work transparently, but that's much more work. We might need to do that at some point, as it's also necessary for std.benchmark and similar code but for now that's probably over-engineering. Here's the revived pull request: https://github.com/D-Programming-Language/dmd/pull/3518 https://github.com/D-Programming-Language/druntime/pull/782
May 01 2014
prev sibling next sibling parent Jeremy Powers via Digitalmars-d <digitalmars-d puremagic.com> writes:
--f46d04426c4619c62e04f85bbbb9
Content-Type: text/plain; charset=UTF-8

 Last but not least, virtually nobody I know runs unittests and then
 main. This is quickly becoming an idiom:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 I think it's time to change that.


The current system of running unit tests prior to main is, in my opinion, fundamentally broken. Logically, the unit tests are a build step - something you do after compile to ensure things are good. Tying them to running main means I cannot have a build that passes unit tests that is also a production build. Granted, it is (as far as I know) impossible to actually compile a production version of code separately from the unittest code, and be able to run the one on the other. But it would be nice to move to something more in line with unittest-as-build-step, rather than -as-different-build. On named tests, I heartily support this. Especially if it comes with the ability to selectively run one test - such is incredibly useful for large projects, to quickly iterate on broken bits. --f46d04426c4619c62e04f85bbbb9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blo= ckquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #c= cc solid;padding-left:1ex"><blockquote class=3D"gmail_quote" style=3D"margi= n:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D"">Last but not least, virtually nobody I know runs unittests = and then<br> main. This is quickly becoming an idiom:<br> <br> version(unittest) void main() {}<br> else void main()<br> {<br> =C2=A0 =C2=A0 ...<br> }<br> <br> I think it&#39;s time to change that.</div></blockquote></blockquote><div>= =C2=A0</div><div><br></div><div>The current system of running unit tests pr= ior to main is, in my opinion, fundamentally broken.</div><div><br></div><d= iv> Logically, the unit tests are a build step - something you do after compile= to ensure things are good. =C2=A0Tying them to running main means I cannot= have a build that passes unit tests that is also a production build.</div>= <div> <br></div><div>Granted, it is (as far as I know) impossible to actually com= pile a production version of code separately from the unittest code, and be= able to run the one on the other. =C2=A0But it would be nice to move to so= mething more in line with unittest-as-build-step, rather than -as-different= -build.</div> <div><br></div><div><br></div><div>On named tests, I heartily support this.= =C2=A0Especially if it comes with the ability to selectively run one test = - such is incredibly useful for large projects, to quickly iterate on broke= n bits.</div> <div><br></div><div><br></div></div></div></div> --f46d04426c4619c62e04f85bbbb9--
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 01 May 2014 12:26:07 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 5/1/14, 12:05 PM, Johannes Pfau wrote:
 Here's the revived pull request:
 https://github.com/D-Programming-Language/dmd/pull/3518
 https://github.com/D-Programming-Language/druntime/pull/782

I'm unclear what this work does even after having read the description. What does it require in addition to just sprinkling unittests around? -- Andrei

Nothing. This just changes the way druntime internally handles the unit tests, but nothing changes for the user: Right now we have one function per module, which calls all unit tests in these modules, but the test runner does not have access to the individual unittest functions. Instead of exposing one function per module, this now exposes every single unittest function. So you can now run every test individually or pass the function pointer to a different thread and run it there. Additionally this provides information about the source location of every unit test. I thought the example is quite clear? ------------- bool tester() { import std.stdio; //iterate all modules foreach(info; ModuleInfo) { //iterate unit test in modules foreach(test; info.unitTests) { //access unittest information writefln("ver=%s file=%s:%s disabled=%s func=%s", test.ver, test.file, test.line, test.disabled, test.func); //execute unittest test.func()(); } } return true; } shared static this() { Runtime.moduleUnitTester = &tester; } ------------- You customize the test runner just like you did before, by setting Runtime.moduleUnitTester in a module constructor. Now you still have to implement a custom test runner to run unittest in parallel, but that's trivial. We can of course add an implementation this to druntime, but we can't use std.parallelism there. What's a little more complicated is the versioning scheme, but that's an implementation detail. It ensures that we can change the exposed information (for example if we want to add a name field, or remove some field) without breaking anything.
May 01 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 01 May 2014 13:00:44 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 Nothing. This just changes the way druntime internally handles the
 unit tests, but nothing changes for the user:

Great, thanks. Just making sure there's no unstated assumptions somewhere. Help with reviewing from the compiler and druntime folks would be appreciated! -- Andrei

I added an parallel test runner example: http://dpaste.dzfl.pl/69baabd83e68 Output: -------- Test 2, Thread 7FE6F0EA4D00 Test 4, Thread 7FE6F0EA4E00 Test 7, Thread 7FE6F0EA4E00 Test 8, Thread 7FE6F0EA4E00 Test 9, Thread 7FE6F0EA4E00 Test 10, Thread 7FE6F0EA4E00 Test 1, Thread 7FE6F0EA4F00 Test 5, Thread 7FE6F0EA4B00 Test 3, Thread 7FE6F0EA4C00 Test 6, Thread 7FE6F0EA4D00
May 01 2014
prev sibling next sibling parent "NVolcz" <volcz kth.se> writes:
The D unittest feature has been a mixed bag from the beginning 
for me.
When a codebase starts to consider to parallelize the unittests 
it's has in many cases become a very expensive to make this 
change. If order of execution was not guaranteed this would force 
coders to make a better long term investment from the beginning.
A nice side effect from having undefined order is that 
programmers are forced to think about state. See 
http://googletesting.blogspot.se/2013/03/testing-on-toilet-testing-state-vs.html

Another link I would like to drop here which is only midly 
relevant. I whish that more developers became aware of the Tests 
Vs. checks discussion.
http://www.satisfice.com/blog/archives/856
May 01 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Thursday, 1 May 2014 at 18:38:15 UTC, w0rp wrote:
 On Thursday, 1 May 2014 at 17:04:53 UTC, Xavier Bigand wrote:
 Le 01/05/2014 16:01, Atila Neves a écrit :
 On Thursday, 1 May 2014 at 11:44:12 UTC, w0rp wrote:
 On Thursday, 1 May 2014 at 11:05:55 UTC, Jacob Carlborg 
 wrote:
 On 2014-04-30 23:35, Andrei Alexandrescu wrote:

 Agreed. I think we should look into parallelizing all 
 unittests. --
 Andrei

I recommend running the tests in random order as well.

This is a bad idea. Tests could fail only some of the time. Even if bugs are missed, I would prefer it if tests did exactly the same thing every time.

They _should_ do exactly the same thing every time. Which is why running in threads or at random is a great way to enforce that. Atila


Tests shouldn't be run in a random order all of the time, perhaps once in a while, manually. Having continuous integration randomly report build failures is crap. Either you should always see a build failure, or you shouldn't see it. You can only test things which are deterministic, at least as far as what you observe. Running tests in a random order should be something you do manually, only when you have some ability to figure out why the tests just failed.

In my experience when a test fails randomly because of ordering, a while loop on the shell running until failure is enough to reproduce it in a few seconds. But as others have mentioned, being able to use a seed to reproduce it exactly is superior. Atila
May 02 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 You've already got it working with version, that's what version 
 is for. Why add yet another way to do it?

Because I'd like something better. It's an idiom that I have used many times (around 15-20 times). I'd like the compiler (or build tool) to avoid me to specify two times what the main module is. Also, the current way to do it, in those modules I have to specify the module name two times (once at the top and once at the bottom, unless I use some compile-time syntheses of the version identifier from the current module name). Bye, bearophile
May 02 2014
prev sibling next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
--001a113a6220188eed04f8a32b19
Content-Type: text/plain; charset=UTF-8

On 1 May 2014 18:40, "Andrei Alexandrescu via Digitalmars-d" <
digitalmars-d puremagic.com> wrote:
 On 5/1/14, 10:32 AM, Brad Anderson wrote:
 It hasn't been C:\TEMP for almost 13 years

About the time when I switched :o). -- Andrei

Amen to that! (Me too) --001a113a6220188eed04f8a32b19 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr"><br> On 1 May 2014 18:40, &quot;Andrei Alexandrescu via Digitalmars-d&quot; &lt;= <a href=3D"mailto:digitalmars-d puremagic.com">digitalmars-d puremagic.com<= /a>&gt; wrote:<br> &gt;<br> &gt; On 5/1/14, 10:32 AM, Brad Anderson wrote:<br> &gt;&gt;<br> &gt;&gt; It hasn&#39;t been C:\TEMP for almost 13 years<br> &gt;<br> &gt;<br> &gt; About the time when I switched :o). -- Andrei</p> <p dir=3D"ltr">Amen to that!=C2=A0 (Me too)</p> --001a113a6220188eed04f8a32b19--
May 05 2014
prev sibling next sibling parent "Messenger" <dont shoot.me> writes:
On Monday, 5 May 2014 at 00:40:41 UTC, Walter Bright wrote:
 D has so many language features, we need a higher bar for 
 adding new ones, especially ones that can be done 
 straightforwardly with existing features.

Sure, but you'll have to agree that there comes a point where library solutions end up being so syntactically convoluted that it becomes difficult to visually parse. Bad-practice non-sensical example. version(ParallelUnittests) const const(TypeTuple!(string, "name", UnittestImpl!SomeT, "test", bool, "result")) testTempfileHammering(string name, alias fun, SomeT, Args...)(Args args) pure safe (TestSuite.standard) if ((name.length > 0) && __traits(parallelizable, fun) && !is(Args) && (Args.length > 2) && allSatisfy!(isSomeString, args)) { /* ... */ }
May 05 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 D has so many language features, we need a higher bar for 
 adding new ones, especially ones that can be done 
 straightforwardly with existing features.

If I am not wrong, all this is needed here is a boolean compile-time flag, like "__is_main_module". I think this is a small enough feature and gives enough back that saves time, to deserve to be a built-in feature. I have needed this for four or five years and the need/desire isn't going away. Bye, bearophile
May 05 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 05 May 2014 10:00:54 +0000
bearophile via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Walter Bright:

 D has so many language features, we need a higher bar for
 adding new ones, especially ones that can be done
 straightforwardly with existing features.

If I am not wrong, all this is needed here is a boolean compile-time flag, like "__is_main_module". I think this is a small enough feature and gives enough back that saves time, to deserve to be a built-in feature. I have needed this for four or five years and the need/desire isn't going away.

As far as I can tell, adding a feature wouldn't add much over simply using a version block for defining your demos. Just because something is done in python does not mean that it is appropriate for D or that it requires adding features to D in order to support it. Though I confess that I'm biased against it, because not only have I never needed the feature that you're looking for, but I'd actually consider it bad practice to organize code that way. It makes no sense to me to make it so that any arbitrary module can be the main module for the program. Such code should be kept separate IMHO. And I suspect that most folks who either haven't done much with python and/or who don't particularly like python would agree with me. Maybe even many of those who use python would; I don't know. Regardless, I'd strongly argue that this is a case where using user-defined versions is the obvious answer. It may not give you what you want, but it gives you want you need in order to make it so that a module has a main that's compiled in only when you want it to be. And D is already quite complicated. New features need to pass a high bar, and adding a feature just so that something is built-in rather than using an existing feature which solves the problem fairly simply definitely does not pass that bar IMHO. I'm completely with Walter on this one. - Jonathan M Davis
May 05 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jonathan M Davis:

 Just because something is done in
 python does not mean that it is appropriate for D or that it 
 requires adding features to D in order to support it.

I agree. On the other hand now I have years of experience in both language and I still have this need in D.
 It makes no sense to me to make it so that
 any arbitrary module can be the main module for the program.

This feature is mostly for single modules, that you can download from archives, the web, etc. So it's for library code contained in single modules. In Python code is usually short, so in a single module you can implement many data structures, data visualization, data converters, etc. So it's quite handy for such modules to have demo, or even an interactive demo. Or they can be used with command lines arguments (with getopt), like a sound file converter. And then you can also import this module from other modules to perform the same operation (like sound file conversion) from your code. So you can use it both as a program that does something, and as a module for a larger system.
 Such code should be kept separate IMHO.

This means that you now have two modules, so to download them atomically you need some kind of packaging, like a zip. If your project is composed by many modules this is not a problem. But if you have a single module project (and this happens often in Python), going from 1 to 2 files is not nice. I have written tens of reusable D modules, and some of them have a demo or are usable stand-alone when you have simpler needs.
 Maybe even many of those who use python would; I don't know.

In Python is a very commonly used idiom. And there is not much in D that makes the same idiom less useful :-) Bye, bearophile
May 05 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 05 May 2014 11:26:29 +0000
bearophile via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Jonathan M Davis:
 Such code should be kept separate IMHO.

This means that you now have two modules, so to download them atomically you need some kind of packaging, like a zip. If your project is composed by many modules this is not a problem. But if you have a single module project (and this happens often in Python), going from 1 to 2 files is not nice. I have written tens of reusable D modules, and some of them have a demo or are usable stand-alone when you have simpler needs.

Honestly, I wouldn't even consider distributing something that was only a single module in size unless it were on the scale of std.datetime, which we've generally agreed is too large for a single module. So, a single module wouldn't have enough functionality to be worth distributing. And even if I were to distribute such a module, I'd let its documentation speak for itself and otherwise just expect the programmer to read the code. Regardless, the version specifier makes it easy to have a version where main is defined for demos or whatever else you might want to do with it. So, I'd suggest just using that. I highly doubt that you'd be able to talk either Walter or Andrei into supporting a separate feature for this. At this point, we're trying to use what we already have to implement new things rather than adding new features to the language, no matter how minor it might seem. New language features are likely be restricted to things where we really need them to be language features. And this doesn't fit that bill. - Jonathan M Davis
May 05 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jonathan M Davis:

 Honestly, I wouldn't even consider distributing something that 
 was only a
 single module in size unless it were on the scale of 
 std.datetime, which we've
 generally agreed is too large for a single module.
 So, a single module
 wouldn't have enough functionality to be worth distributing.

This reasoning style is similar to the Groucho Marx quote: "I don't care to belong to any club that will have me as a member" In the Python world online you can find thousands of single-module projects (few of them are mine). I have plenty of single D modules that encapsulate a single functionality. In Haskell cabal you can find many single modules that add functionality (plus larger projects like Diagrams). And I think D has to strongly encourage the creation of such ecosystem of modules that you download and use in your programs. You can't have everything in the standard library, it's not wise to re-write them (like 2D vectors, I have already seen them implemented ten different times in the D world), and there are plenty of useful things that can be contained in single modules, especially if such modules can import the functionality of one or more other modules.
 And even if I were to distribute such a module, I'd let its
 documentation speak for itself
 and otherwise just expect the programmer to read the code.

A demo and the documentation are both useful. And the documentation can't replace stand-alone functionality.
 Regardless, the version specifier makes it easy to have a 
 version where main is defined for demos or whatever else
 you might want to do with it.

Bye, bearophile
May 05 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
However, the community is starting to standardize around Dub as 
the standard package manager. Dub makes downloading a package as 
easy as editing a JSON file (and it scales such that you can 
download a project of any size this way). Did Python have a 
proper package manager before this idiom arose?
May 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 May 2014 at 16:08:23 UTC, Andrei Alexandrescu 
wrote:
 It got full because of tests (surprise!). Your actions?

Fix the machine and reduce the output created by the unittests. It's a simple engineering problem. -- Andrei

You can't. You have not control over that machine you don't even exactly know that test has failed because of full /tmp/ - all you got is a bug report that can't be reproduced on your machine. It is not that simple already and it can get damn complicated once you get to something like network I/O
May 05 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Meta:

 However, the community is starting to standardize around Dub as 
 the standard package manager. Dub makes downloading a package 
 as easy as editing a JSON file (and it scales such that you can 
 download a project of any size this way).

Having package manager(s) in Python doesn't make single module Python projects less popular or less appreciated. Most Python projects are very small, thanks to both standard library and code succinctness (that allows a small program to do a lot), and the presence of an healthy ecology of third party modules that you can import to avoid re-writing things already done by other people. All this should become more common in the D world :-)
 Did Python have a proper package manager before this idiom 
 arose?

Both are very old, and I am not sure, but I think the main module idiom is precedent. Bye, bearophile
May 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 5 May 2014 at 15:36:19 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 8:11 AM, Dicebot wrote:
 On Thursday, 1 May 2014 at 16:08:23 UTC, Andrei Alexandrescu 
 wrote:
 It got full because of tests (surprise!). Your actions?

Fix the machine and reduce the output created by the unittests. It's a simple engineering problem. -- Andrei

You can't. You have not control over that machine you don't even exactly know that test has failed because of full /tmp/ - all you got is a bug report that can't be reproduced on your machine. It is not that simple already and it can get damn complicated once you get to something like network I/O

I know, incidentally the hhvm team has had the same problem two weeks ago. They fixed it (wthout removing file I/O from unittests). It's fixable. That's it.

It is possible to write a unit test which provides graceful failure reporting for such issues but once you get there it becomes hard to see actual tests behind boilerplate of environmental verification and actual application code behind tests. Any tests that rely on I/O need some sort of commonly repeated initialize-verify-test-finalize pattern, one that is simply impractical to do with unit tests.
 This segment started with your claim that unittests should do 
 no file I/O because they may fail with a full /tmp/. I disagree 
 with that, and with framing the full /tmp/ problem as a problem 
 with the unittests doing file I/O.

It was just a most simple example. "Unittests should do no I/O because any sort of I/O can fail because of reasons you don't control from the test suite" is an appropriate generalization of my statement. Full /tmp is not a problem, there is nothing broken about system with full /tmp. Problem is test reporting that is unable to connect failure with /tmp being full unless you do environment verification.
May 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 5 May 2014 at 16:33:42 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 8:55 AM, Dicebot wrote:
 It was just a most simple example. "Unittests should do no I/O 
 because
 any sort of I/O can fail because of reasons you don't control 
 from the
 test suite" is an appropriate generalization of my statement.

 Full /tmp is not a problem, there is nothing broken about 
 system with
 full /tmp. Problem is test reporting that is unable to connect 
 failure
 with /tmp being full unless you do environment verification.

Different strokes for different folks. -- Andrei

There is nothing subjective about it. It is a very well-define practical goal - getting either reproducible or informative reports for test failures from machines you don't have routine access to. Why still keeping test sources maintainable (ok this part is subjective). It is relatively simple engineering problem but you discard widely adopted solution for it (strict control of test requirements) without proposing any real alternative. "I will yell at someone when it breaks" is not really a solution.
May 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 5 May 2014 at 18:24:43 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 10:08 AM, Dicebot wrote:
 On Monday, 5 May 2014 at 16:33:42 UTC, Andrei Alexandrescu 
 wrote:
 On 5/5/14, 8:55 AM, Dicebot wrote:
 It was just a most simple example. "Unittests should do no 
 I/O because
 any sort of I/O can fail because of reasons you don't 
 control from the
 test suite" is an appropriate generalization of my statement.

 Full /tmp is not a problem, there is nothing broken about 
 system with
 full /tmp. Problem is test reporting that is unable to 
 connect failure
 with /tmp being full unless you do environment verification.

Different strokes for different folks. -- Andrei

There is nothing subjective about it.

Of course there is. -- Andrei

You are not helping your point to look reasonable.
May 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 5 May 2014 at 18:29:40 UTC, Andrei Alexandrescu wrote:
 My understanding here is you're trying to make dogma out of 
 engineering choices that may vary widely across projects and 
 organizations. No thanks.

 Andrei

I am asking to either suggest an alternative solution or to clarify why you don't consider it is an important problem. Dogmatic approach that solves the issue is still better than ignoring it completely. Right now I am afraid you will push for quick changes that will reduce elegant simplicity of D unittest system without providing a sound replacement that will actually fit into more ambitious use cases (as whole "parallel" thing implies).
May 05 2014
prev sibling next sibling parent reply Bruno Medeiros <bruno.do.medeiros+dng gmail.com> writes:
On 30/04/2014 16:43, Andrei Alexandrescu wrote:
 Hello,


 A coworker mentioned the idea that unittests could be run in parallel
 (using e.g. a thread pool).

There has been a lot of disagreement in this discussion of whether "unittests" blocks should run in parallel or not. Not everyone is agreeing with Andrei's idea in the first place. I am another in such position. True, like Dicebot, Russel, and others mentioned, a Unit Test should be a procedure with no side-effects (or side-effects independent from the other Unit Tests), and as such, able to run in parallel. Otherwise they are an Integration Test. But before we continue the discussion, we are missing am more basic assumption here: Do we want D to have a Unit-Testing facility, or a Testing facility?? In other words, do we want to be able to write automated tests that are Integration tests or just Unit tests? Because if we go with this option of making D unittest blocks run in parallel, we kill the option of them supporting Integration Tests. I don't think this is good. *Unit testing frameworks in other languages (JUnit for Java for example), provide full support for Integration tests, despite the "Unit" in their names. This is good. I think Integration tests are much more common than in "real-world" applications than people give credit for. Personally I find the distinction between Unit tests and Integrations tests not very useful in practice. It is accurate, but not very useful. In my mental model I don't make a distinction. I write a test that tests a component, or part of a component. The component might be a low-level component that depends on little or no other components - then I have a Unit test. Or it might be a higher level component that depends on other components (which might need to mocked in the test) - then I have an Integration test. But they are not different enough that a different framework should be necessary to write each of them. -- Bruno Medeiros https://twitter.com/brunodomedeiros
May 06 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 6 May 2014 at 15:54:30 UTC, Bruno Medeiros wrote:
 But before we continue the discussion, we are missing am more 
 basic assumption here: Do we want D to have a Unit-Testing 
 facility, or a Testing facility?? In other words, do we want to 
 be able to write automated tests that are Integration tests or 
 just Unit tests? Because if we go with this option of making D 
 unittest blocks run in parallel, we kill the option of them 
 supporting Integration Tests. I don't think this is good.

These days I often find myself leaning towards writing mostly integration tests with only limited amount of unit tests. But writing good integration test is very different from writing good unit test and usually implies quite lot of boilerplate. Truth is D does not currently have any higher-level facility at all. It has an _awesome_ unit test facility which gets often misused for writing sloppy integration tests. I'd love to keep existing facility as-is and think about providing good library augmentation for any sort of higher level approach. Key property of D unit tests is how easy it is to add those inline to existing project, unconstrained simplicity. In perfect world those are closer to contracts than other types of tests. This provides good basic sanity check for all modules you recursively import when run via `rdmd -unittest`. Good integration test is very different. It has certain assumptions about initial system state and notifies user if those are not met. It can take ages to run and can test real-world situations. It is not supposed to be run implicitly and frequently. You don't want to keep your integration tests inline because of amount of boilerplate code those usually need. I see no good in trying to unite those very different beasts and my experience with existing test libraries has been very unpleasant in that regard.
May 06 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-06 19:58, Dicebot wrote:

 These days I often find myself leaning towards writing mostly
 integration tests with only limited amount of unit tests. But writing
 good integration test is very different from writing good unit test and
 usually implies quite lot of boilerplate. Truth is D does not currently
 have any higher-level facility at all. It has an _awesome_ unit test
 facility which gets often misused for writing sloppy integration tests.

 I'd love to keep existing facility as-is and think about providing good
 library augmentation for any sort of higher level approach.

 Key property of D unit tests is how easy it is to add those inline to
 existing project, unconstrained simplicity. In perfect world those are
 closer to contracts than other types of tests. This provides good basic
 sanity check for all modules you recursively import when run via `rdmd
 -unittest`.

 Good integration test is very different. It has certain assumptions
 about initial system state and notifies user if those are not met. It
 can take ages to run and can test real-world situations. It is not
 supposed to be run implicitly and frequently. You don't want to keep
 your integration tests inline because of amount of boilerplate code
 those usually need.

 I see no good in trying to unite those very different beasts and my
 experience with existing test libraries has been very unpleasant in that
 regard.

I don't see why would be bad to use "unittest" for integration tests, except for the misguided name. It's perfectly to place "unittest" is completely different modules and packages. They don't need to be placed inline. I see it as a code place to but code for testing. Then I don't have to come up with awkward names for regular functions. It's also a good palace since D doesn't allow to place statements and expression at module level. Sure there's module constructors but I don't think that's any better. -- /Jacob Carlborg
May 06 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 06/05/14 20:39, Dicebot wrote:
 On Tuesday, 6 May 2014 at 18:28:27 UTC, Jacob Carlborg wrote:
 d.
 I don't see why would be bad to use "unittest" for integration tests,
 except for the misguided name. It's perfectly to place "unittest" is
 completely different modules and packages. They don't need to be
 placed inline.

Well I am actually guilty of doing exactly that because it allows me to merge coverage analysis files :) But it is not optimal situation once you consider something like parallel tests as compiler does not know which of those blocks are "true" unit tests. It also makes difficult to define a common "idiomatic" way to organize testing of D projects. I'd also love to see a test library that helps with defining integration tests structure (named tests grouped by common environment requirements doing automatic cleanup upon finishing the group/block) without resorting to custom classes AND without interfering with simplicity of existing unittests. I think it all can be done by keeping existing single "unittest" keyword but using various annotations. Then integration tests can be done as separate application that uses imaginary Phobos integration tests library to interpret those annotations and provide more complex test structure. And running plain `rdmd -unittest` on actual application modules will still continue to do the same good old thing.

So you're saying to use the "unittest" keyword but with a UDA? Something I already do, but for unit tests. Well my idea for a testing framework would work both for unit tests and other, higher levels of test. describe("toMsec") { it("returns the time in milliseconds") unittest { assert(true); } } -- /Jacob Carlborg
May 06 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 07/05/14 16:05, Dicebot wrote:

 Have never liked that fancy description syntax of "smart" testing
 frameworks.

I hate plain unit test blocks with just a bunch of asserts. It's impossible to know that's being tested. -- /Jacob Carlborg
May 07 2014
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-05-07 17:05, H. S. Teoh via Digitalmars-d wrote:

 Huh? Isn't that what unittest blocks are about? To verify that certain
 assumed conditions are actually true at runtime?

 Verbal descriptions can be put in comments, if need be, can't they?

What Atila said. -- /Jacob Carlborg
May 07 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-08 14:56, Dicebot wrote:

 You don't need artificial pseudo syntax for that.
 assert!("==") + named tests is good enough to get the context and for
 detailed investigation you need file and line number anyway. Stuff like
 RSpec is extreme opposite of KISS.

RSpec uses a syntax that makes it easier to read a test. To understand what it actually tests. I mean, what the h*ll does this unit test tests: https://github.com/D-Programming-Language/phobos/blob/master/std/numeric.d#L995 I'm mostly interested in the describe/it functionality, not the fancy asserts, although I don't mind them either. describe("foo") { it("should do something useful") unittest { } } Is not so much different from what you suggested with named unit tests. -- /Jacob Carlborg
May 08 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-05-09 13:57, Dicebot wrote:

 This is redundant as D unittest blocks are associated with symbols they
 are placed next to.

I prefer to keep my tests in a separate directory.
 It introduces bunch of artificial annotations for something that can be
 taken care of by a single attribute as a side effect. Not KISS.

I just don't agree. It's a bit hard to do in D, but in the Ruby version, for each "describe" block an anonymous class is created. Nested blocks inherits from the outer blocks. The "it" blocks are evaluated inside an instance of the closest "describe" block. This makes it very nice to setup data for the tests. You can have helper methods in the "describe" blocks, overriding methods in the outer blocks and so on, very convenient. As a bonus, you can run the test with a special formatter. This will print all strings passed to the "describe" and "it" blocks in a structured way. Suddenly you can generate documentation of how you're system is supposed to work. -- /Jacob Carlborg
May 10 2014
prev sibling parent Bruno Medeiros <bruno.do.medeiros+dng gmail.com> writes:
On 06/05/2014 18:58, Dicebot wrote:
 On Tuesday, 6 May 2014 at 15:54:30 UTC, Bruno Medeiros wrote:
 But before we continue the discussion, we are missing am more basic
 assumption here: Do we want D to have a Unit-Testing facility, or a
 Testing facility?? In other words, do we want to be able to write
 automated tests that are Integration tests or just Unit tests? Because
 if we go with this option of making D unittest blocks run in parallel,
 we kill the option of them supporting Integration Tests. I don't think
 this is good.

These days I often find myself leaning towards writing mostly integration tests with only limited amount of unit tests. But writing good integration test is very different from writing good unit test and usually implies quite lot of boilerplate. Truth is D does not currently have any higher-level facility at all. It has an _awesome_ unit test facility which gets often misused for writing sloppy integration tests.

Indeed: I also find myself writing more integration tests than unit tests, at least in the way I consider an integration tests to be (in some cases the distinction between integration and unit test may not be very easy or clear, IMO).
 I'd love to keep existing facility as-is and think about providing good
 library augmentation for any sort of higher level approach.

The unittest block is enough right now to support integration tests. To support common test fixture setup (akin to Before and After in xUnit), perhaps some syntax sugar could be added, although with D's language facilities (meta-programming, functional constructs, scope statements, etc.), we can do pretty well with existing functionality already.
 Good integration test is very different. It has certain assumptions
 about initial system state and notifies user if those are not met. It
 can take ages to run and can test real-world situations. It is not
 supposed to be run implicitly and frequently. You don't want to keep
 your integration tests inline because of amount of boilerplate code
 those usually need.

They are somewhat different. I wouldn't say very different. I don't agree integration tests usually take ages to run. Some of them can run fairly fast too, and are executed as frequently as unit tests. In DDT for example I always run unit tests the same time as integration tests. As I said, I don't find it useful to have a strict distinction between those. Rather, if I want to run a subset of tests, usually what I filter on is running only the tests of a certain plugin (DDT has 3 plugins), Java package, or Java class. Additionally the parser tests can be run in what I call "Lite" mode, which instead of running the full test suite, skips some of the heavyweight, parameterized tests, to make the suite run faster. Most of the cases are generated from templates, others are blind mass parse tests, such as parsing all source modules in Phobos. But what "Lite" mode cuts is not Integration tests, but rather the input sets of parameterized tests. As for keeping integration tests inline or not, yeah, you are likely to prefer putting them in a separate file. Doesn't mean we need a different language construct other than the unittest block for that.
 I see no good in trying to unite those very different beasts and my
 experience with existing test libraries has been very unpleasant in that
 regard.

What test libraries/frameworks have you used? -- Bruno Medeiros https://twitter.com/brunodomedeiros
May 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 6 May 2014 at 18:28:27 UTC, Jacob Carlborg wrote:
d.
 I don't see why would be bad to use "unittest" for integration 
 tests, except for the misguided name. It's perfectly to place 
 "unittest" is completely different modules and packages. They 
 don't need to be placed inline.

Well I am actually guilty of doing exactly that because it allows me to merge coverage analysis files :) But it is not optimal situation once you consider something like parallel tests as compiler does not know which of those blocks are "true" unit tests. It also makes difficult to define a common "idiomatic" way to organize testing of D projects. I'd also love to see a test library that helps with defining integration tests structure (named tests grouped by common environment requirements doing automatic cleanup upon finishing the group/block) without resorting to custom classes AND without interfering with simplicity of existing unittests. I think it all can be done by keeping existing single "unittest" keyword but using various annotations. Then integration tests can be done as separate application that uses imaginary Phobos integration tests library to interpret those annotations and provide more complex test structure. And running plain `rdmd -unittest` on actual application modules will still continue to do the same good old thing.
May 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 7 May 2014 at 06:34:44 UTC, Jacob Carlborg wrote:
 So you're saying to use the "unittest" keyword but with a UDA?

I think this is most reasonable compromise that does not harm existing system.
 Something I already do, but for unit tests. Well my idea for a 
 testing framework would work both for unit tests and other, 
 higher levels of test.

  describe("toMsec")
 {
      it("returns the time in milliseconds") unittest
     {
         assert(true);
     }
 }

Which is exactly why I'd like to defer exact annotation to library solution - exact requirements for such framework are very different. I'd want to see something like this instead: name("Network test 2") requires("Network test 1") cleanup!removeTemporaries unittest { // do stuff } Have never liked that fancy description syntax of "smart" testing frameworks.
May 07 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, May 07, 2014 at 04:55:25PM +0200, Jacob Carlborg via Digitalmars-d
wrote:
 On 07/05/14 16:05, Dicebot wrote:
 
Have never liked that fancy description syntax of "smart" testing
frameworks.

I hate plain unit test blocks with just a bunch of asserts. It's impossible to know that's being tested.

Huh? Isn't that what unittest blocks are about? To verify that certain assumed conditions are actually true at runtime? Verbal descriptions can be put in comments, if need be, can't they? T -- The right half of the brain controls the left half of the body. This means that only left-handed people are in their right mind. -- Manoj Srivastava
May 07 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 7 May 2014 at 15:07:20 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 On Wed, May 07, 2014 at 04:55:25PM +0200, Jacob Carlborg via 
 Digitalmars-d wrote:
 On 07/05/14 16:05, Dicebot wrote:
 
Have never liked that fancy description syntax of "smart" 
testing
frameworks.

I hate plain unit test blocks with just a bunch of asserts. It's impossible to know that's being tested.

Huh? Isn't that what unittest blocks are about? To verify that certain assumed conditions are actually true at runtime? Verbal descriptions can be put in comments, if need be, can't they?

They can. But those descriptions are not included in failing test output. What I think Jacob might be getting to as well is that assertEquals or the more RSpec-like "foo.should equal 3" is more readable than the raw asserts. The context matters. In some frameworks that means using test names like testThatWhenIDoThisThenTheOtherThingActuallyHappens (which we'd get if we can have named unit tests), RSpec tries to be more readable but in the end it's all about: 1) Documenting what the code is supposed to do 2) Knowing what test failed and what it was testing Atila
May 07 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 7 May 2014 at 16:09:28 UTC, Atila Neves wrote:
 They can. But those descriptions are not included in failing 
 test output. What I think Jacob might be getting to as well is 
 that assertEquals or the more RSpec-like "foo.should equal 3" 
 is more readable than the raw asserts.

 The context matters. In some frameworks that means using test 
 names like testThatWhenIDoThisThenTheOtherThingActuallyHappens 
 (which we'd get if we can have named unit tests), RSpec tries 
 to be more readable but in the end it's all about:

 1) Documenting what the code is supposed to do
 2) Knowing what test failed and what it was testing

You don't need artificial pseudo syntax for that. assert!("==") + named tests is good enough to get the context and for detailed investigation you need file and line number anyway. Stuff like RSpec is extreme opposite of KISS.
May 08 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 7 May 2014 at 14:34:41 UTC, Bruno Medeiros wrote:
 On 06/05/2014 18:58, Dicebot wrote:
 I see no good in trying to unite those very different beasts 
 and my
 experience with existing test libraries has been very 
 unpleasant in that
 regard.

What test libraries/frameworks have you used?

I have C/C++ origins so it was mostly stuff like cppUnit, xUnit and Boost one as far as I can remember.
May 08 2014
prev sibling next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 2014-05-08 at 13:03 +0000, Dicebot via Digitalmars-d wrote:
[…]
 I have C/C++ origins so it was mostly stuff like cppUnit, xUnit 
 and Boost one as far as I can remember.

The current C++ test framework front runner is probably Phil Nash's Catch https://github.com/philsquared/Catch -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
May 08 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 8 May 2014 at 18:54:30 UTC, Jacob Carlborg wrote:
 I mean, what the h*ll does this unit test tests:

 https://github.com/D-Programming-Language/phobos/blob/master/std/numeric.d#L995

It is explained in comments there. And it won't become more simple if you add some fancy syntax there. It looks complicated because it _is_ complicated, not because syntax is bad.
  describe("foo")

This is redundant as D unittest blocks are associated with symbols they are placed next to.
 {
      it("should do something useful") unittest {

This is essentially name with an overly smart name and weird attribute placement.
 Is not so much different from what you suggested with named 
 unit tests.

It introduces bunch of artificial annotations for something that can be taken care of by a single attribute as a side effect. Not KISS.
May 09 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 5 May 2014 at 18:58:37 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 11:47 AM, Dicebot wrote:
 On Monday, 5 May 2014 at 18:29:40 UTC, Andrei Alexandrescu 
 wrote:
 My understanding here is you're trying to make dogma out of
 engineering choices that may vary widely across projects and
 organizations. No thanks.

 Andrei

I am asking to either suggest an alternative solution or to clarify why you don't consider it is an important problem.

"Clean /tmp/ judiciously."

This is solution for "failing test" problem. Problem I speak about is "figuring out why test has failed".
 The problem with your stance, i.e.:

 "Unittests should do no I/O because any sort of I/O can fail 
 because
 of reasons you don't control from the test suite" is an 
 appropriate
 generalization of my statement.

is that it immediately generalizes into the unreasonable: "Unittests should do no $X because any sort of $X can fail because of reasons you don't control from the test suite". So that gets into machines not having any memory available, with full disks etc.

It is great you have mentioned RAM here as it nicely draws a border-line. Being out of memory throws specific Error which is unlikely to be caught and clearly identifies problem. Disk I/O failure throws Exception which can be easily consumed somewhere inside tested control flow resulting in absolutely mysterious test failures. It is borderline of Error vs Exception - fatal problem incompatible with further execution and routine problem application is expected to handle.
 Just make sure test machines are prepared for running unittests 
 to the extent unittests are expecting them to. We're wasting 
 time trying to frame this as a problem purely related to 
 unittests alone.

Again: you don't have control of test machines for something like language standard library. It is not purely unittest problem, it is problem hard to solve staying within infrastructure of unittests.
May 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 6 May 2014 at 18:13:01 UTC, Andrei Alexandrescu wrote:
 On 5/6/14, 10:43 AM, Dicebot wrote:
 Disk I/O failure throws Exception which can be easily consumed 
 somewhere
 inside tested control flow resulting in absolutely mysterious 
 test failures.

If you're pointing out full /tmp/ should be nicely diagnosed by the unittest, I agree. -- Andrei

Good we have a common base ground :) It inevitably arises next question though : how unittest can diagnose it? Catching exception is not always possible as it can be consumed inside tested function resulting in different observable behavior when /tmp/ is full. I can't imagine anything better than verifying /tmp/ is not full before running bunch of tests. Will you agree with this one too?
May 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 6 May 2014 at 20:41:01 UTC, Andrei Alexandrescu wrote:
 Fail with diagnostic. -- Andrei

..and do that for every single test case which is affected. Which requires either clear test execution order (including cross-module test dependencies) or shared boilerplate (which becomes more messy if more environment needs to tested). Something that is not nicely supported by built-in construct.
May 07 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 6 May 2014 at 20:41:01 UTC, Andrei Alexandrescu wrote:
 On 5/6/14, 11:27 AM, Dicebot wrote:
 On Tuesday, 6 May 2014 at 18:13:01 UTC, Andrei Alexandrescu 
 wrote:
 On 5/6/14, 10:43 AM, Dicebot wrote:
 Disk I/O failure throws Exception which can be easily 
 consumed somewhere
 inside tested control flow resulting in absolutely 
 mysterious test
 failures.

If you're pointing out full /tmp/ should be nicely diagnosed by the unittest, I agree. -- Andrei

Good we have a common base ground :) It inevitably arises next question though : how unittest can diagnose it?

Fail with diagnostic. -- Andrei

This is funnily ambiguous statement :)
May 08 2014