www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Named unittests

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
We're having a strong need for named unittests at Facebook for multiple 
reasons.

1. We have sophisticated tooling that verifies whether unittests are 
flaky. The automated monitor (for e.g. C++) figures whether a given 
unittest fails several times across several commits. Unittests are 
identified by name; relying on file/line is impossible because the line 
of a failure is not stable across changes.

2. Again for efficient automated testing and flakiness detection, one 
should be able to run only a subset of unittests by mentioning them by 
line in the command line. Note that this implies there's no 
interdependency between distinct unittests, which is fine because the 
new ability is opt-on; I'd say is pure style anyway.

3. Mentioning unittest names in failure messages helps human 
communication (e.g. "AddPeer is failing after your change"). This is 
impossible with file and line numbers.

I'd like to make a DIP for named unittests. Who can help me with that?


Andrei
Mar 30 2015
next sibling parent reply "Panke" <tobias pankrath.net> writes:
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:
 We're having a strong need for named unittests at Facebook for 
 multiple reasons.

 1. We have sophisticated tooling that verifies whether 
 unittests are flaky. The automated monitor (for e.g. C++) 
 figures whether a given unittest fails several times across 
 several commits. Unittests are identified by name; relying on 
 file/line is impossible because the line of a failure is not 
 stable across changes.

 2. Again for efficient automated testing and flakiness 
 detection, one should be able to run only a subset of unittests 
 by mentioning them by line in the command line. Note that this 
 implies there's no interdependency between distinct unittests, 
 which is fine because the new ability is opt-on; I'd say is 
 pure style anyway.

 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). 
 This is impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei
I've implemented this in a library and I'm sure others have as well. Are you sure, you want a language solution?
Mar 30 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/15 2:55 PM, Panke wrote:
 I've implemented this in a library and I'm sure others have as well. Are
 you sure, you want a language solution?
With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
Mar 30 2015
next sibling parent "Panke" <tobias pankrath.net> writes:
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
wrote:
 On 3/30/15 2:55 PM, Panke wrote:
 I've implemented this in a library and I'm sure others have as 
 well. Are
 you sure, you want a language solution?
With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
In my case: no. I wrote it before UDA and wanted to be able to generate testcases at runtime. Used this to register a test case for a SAT solver for every instance in a subdirectory etc.
Mar 30 2015
prev sibling parent reply "tcak" <tcak gmail.com> writes:
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
wrote:
 On 3/30/15 2:55 PM, Panke wrote:
 I've implemented this in a library and I'm sure others have as 
 well. Are
 you sure, you want a language solution?
With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
The "unittest" is a language thing. Why would name support be put into library I don't get it. unittest{} => Unnamed unittest!"Testing new classes" {} => Named (Explained)
Mar 31 2015
parent reply "w0rp" <devw0rp gmail.com> writes:
On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:
 On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
 wrote:
 On 3/30/15 2:55 PM, Panke wrote:
 I've implemented this in a library and I'm sure others have 
 as well. Are
 you sure, you want a language solution?
With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
The "unittest" is a language thing. Why would name support be put into library I don't get it. unittest{} => Unnamed unittest!"Testing new classes" {} => Named (Explained)
Becaused adding language features takes longer than using a library, and every single feature, no matter how seemingly simple, will increase the number of langauge bugs and lead to more odd things happening.
Mar 31 2015
parent "tcak" <tcak gmail.com> writes:
On Tuesday, 31 March 2015 at 12:10:15 UTC, w0rp wrote:
 On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:
 On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
 wrote:
 On 3/30/15 2:55 PM, Panke wrote:
 I've implemented this in a library and I'm sure others have 
 as well. Are
 you sure, you want a language solution?
With attributes? That might be palatable but only as a standard solution. I'd want to add that to Phobos. -- Andrei
The "unittest" is a language thing. Why would name support be put into library I don't get it. unittest{} => Unnamed unittest!"Testing new classes" {} => Named (Explained)
Becaused adding language features takes longer than using a library, and every single feature, no matter how seemingly simple, will increase the number of langauge bugs and lead to more odd things happening.
I can understand this, and I agree with you. But this is no different then ordinary businesses those do not care about quality of their products or customer service as long as they make profit. And as you know, we mostly swear at them due to this.
Mar 31 2015
prev sibling parent reply "Panke" <tobias pankrath.net> writes:
 I've implemented this in a library and I'm sure others have as 
 well. Are you sure, you want a language solution?
Basic principle: --- unittest { testCase("a testcase", { // actual test code here }); } --- Testcase registers the structure and the runtime just runs the unittest as is does today and than calls all closures registered during the unittest. Makes this fully backward compatible.
Mar 30 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/15 2:59 PM, Panke wrote:
 I've implemented this in a library and I'm sure others have as well.
 Are you sure, you want a language solution?
Basic principle: --- unittest { testCase("a testcase", { // actual test code here }); } ---
Looks a bit too much work.
 Testcase registers the structure and the runtime just runs the unittest
 as is does today and than calls all closures registered during the
 unittest. Makes this fully backward compatible.
Backward compatibility is not an issue seeing this is a pure addition. Andrei
Mar 30 2015
parent "Panke" <tobias pankrath.net> writes:
 Backward compatibility is not an issue seeing this is a pure 
 addition.


 Andrei
I mean that it lives peacefully alongside normal unittest blocks.
Mar 30 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
I'd prefer putting alternative test runner into Phobos instead 
which will support ` name("Something") unittest { }`
Mar 30 2015
next sibling parent "Brian Schott" <briancschott gmail.com> writes:
On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:
 I'd prefer putting alternative test runner into Phobos instead 
 which will support ` name("Something") unittest { }`
I agree that an annotation (Probably defined in object.d) is the best way to handle this. Here's something I don't think many people know: You can't use `~` in deprecated() attributes. You can only put a string literal in the parenthesis. If you want to break messages over multiple lines you need to rely on implicit string concatenation.
Mar 30 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/30/15 5:58 PM, Dicebot wrote:
 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I asked for module RTInfo -- we can use this information in the runtime to instrument how we run unit tests. We still don't have module RTInfo. And yes, then it can be a library solution. unittests are a language feature, but only in how they are compiled and linked. The runtime is fully responsible for how they are run. All we need is a way to tell the compiler how to describe them to the runtime. -Steve
Mar 31 2015
parent reply "w0rp" <devw0rp gmail.com> writes:
On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer 
wrote:
 On 3/30/15 5:58 PM, Dicebot wrote:
 I'd prefer putting alternative test runner into Phobos instead 
 which
 will support ` name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I asked for module RTInfo -- we can use this information in the runtime to instrument how we run unit tests. We still don't have module RTInfo. And yes, then it can be a library solution. unittests are a language feature, but only in how they are compiled and linked. The runtime is fully responsible for how they are run. All we need is a way to tell the compiler how to describe them to the runtime. -Steve
ModuleInfo does actually exist, but it's not documented. I'm not sure if it's usable for this purpose though. Maybe?
Mar 31 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/31/15 9:05 AM, w0rp wrote:
 On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:
 On 3/30/15 5:58 PM, Dicebot wrote:
 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I asked for module RTInfo -- we can use this information in the runtime to instrument how we run unit tests. We still don't have module RTInfo. And yes, then it can be a library solution. unittests are a language feature, but only in how they are compiled and linked. The runtime is fully responsible for how they are run. All we need is a way to tell the compiler how to describe them to the runtime.
ModuleInfo does actually exist, but it's not documented. I'm not sure if it's usable for this purpose though. Maybe?
No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271 Essentially, you have user-defined generation of runtime info stored inside the ModuleInfo. When this is working, we can do whatever we want for unit tests via attributes. -Steve
Mar 31 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/31/15 9:28 AM, Steven Schveighoffer wrote:
 On 3/31/15 9:05 AM, w0rp wrote:
 On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:
 On 3/30/15 5:58 PM, Dicebot wrote:
 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`
Yes, this is one of the benefits I touted 2 years ago when I asked for module RTInfo -- we can use this information in the runtime to instrument how we run unit tests. We still don't have module RTInfo. And yes, then it can be a library solution. unittests are a language feature, but only in how they are compiled and linked. The runtime is fully responsible for how they are run. All we need is a way to tell the compiler how to describe them to the runtime.
ModuleInfo does actually exist, but it's not documented. I'm not sure if it's usable for this purpose though. Maybe?
No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271 Essentially, you have user-defined generation of runtime info stored inside the ModuleInfo. When this is working, we can do whatever we want for unit tests via attributes.
See here too: https://issues.dlang.org/show_bug.cgi?id=10023 -Steve
Mar 31 2015
prev sibling parent reply "Meta" <jared771 gmail.com> writes:
On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:
 I'd prefer putting alternative test runner into Phobos instead 
 which will support ` name("Something") unittest { }`
Aren't unittest blocks just special functions? If that's the case, there should be no problem being able to give them names. It seems to me that it would entail the lifting of a restriction rather than a real language change. Before: unittest { assert(1 == 1); } After: unittest checkBasicLaw { assert(1 == 1); }
Mar 31 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 7:55 AM, Meta wrote:
 On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:
 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`
Aren't unittest blocks just special functions? If that's the case, there should be no problem being able to give them names. It seems to me that it would entail the lifting of a restriction rather than a real language change. Before: unittest { assert(1 == 1); } After: unittest checkBasicLaw { assert(1 == 1); }
That's what I had in mind, too. -- Andrei
Mar 31 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-31 16:55, Meta wrote:

 Aren't unittest blocks just special functions? If that's the case, there
 should be no problem being able to give them names. It seems to me that
 it would entail the lifting of a restriction rather than a real language
 change.

 Before:
 unittest
 {
      assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
      assert(1 == 1);
 }
I prefer a UDA accepting a string, this can contain spaces and it's not limited to identifier names. -- /Jacob Carlborg
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 1:04 PM, Jacob Carlborg wrote:
 On 2015-03-31 16:55, Meta wrote:

 Aren't unittest blocks just special functions? If that's the case, there
 should be no problem being able to give them names. It seems to me that
 it would entail the lifting of a restriction rather than a real language
 change.

 Before:
 unittest
 {
      assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
      assert(1 == 1);
 }
I prefer a UDA accepting a string, this can contain spaces and it's not limited to identifier names.
I used to think the same, but then I figured a bit of structure might be preferable. -- Andrei
Mar 31 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 20:18:41 UTC, Andrei Alexandrescu 
wrote:
 On 3/31/15 1:04 PM, Jacob Carlborg wrote:
 On 2015-03-31 16:55, Meta wrote:

 Aren't unittest blocks just special functions? If that's the 
 case, there
 should be no problem being able to give them names. It seems 
 to me that
 it would entail the lifting of a restriction rather than a 
 real language
 change.

 Before:
 unittest
 {
     assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
     assert(1 == 1);
 }
I prefer a UDA accepting a string, this can contain spaces and it's not limited to identifier names.
I used to think the same, but then I figured a bit of structure might be preferable. -- Andrei
I see no value in test names limited to valid identifiers. It is only tiny bit more informative than `unittestXXX` we have already. If we add names, please, let them be proper names that are easy to read.
Mar 31 2015
parent "krzaq" <dlangmailinglist krzaq.cc> writes:
On Tuesday, 31 March 2015 at 21:24:20 UTC, Dicebot wrote:
 I see no value in test names limited to valid identifiers. It 
 is only tiny bit more informative than `unittestXXX` we have 
 already. If we add names, please, let them be proper names that 
 are easy to read.
I'd rather have the name obey the same restrictions as normal function names and keep additional information in optional description("...")
Apr 02 2015
prev sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 20:04:01 UTC, Jacob Carlborg wrote:
 On 2015-03-31 16:55, Meta wrote:

 Aren't unittest blocks just special functions? If that's the 
 case, there
 should be no problem being able to give them names. It seems 
 to me that
 it would entail the lifting of a restriction rather than a 
 real language
 change.

 Before:
 unittest
 {
     assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
     assert(1 == 1);
 }
I prefer a UDA accepting a string, this can contain spaces and it's not limited to identifier names.
Limiting unittest names to legal identifiers will save a lot of headache when we set our tools to actually use these names. Matching a legal identifier in a text stream is much easier than matching an arbitrary unicode string, even if that string is escaped.
Mar 31 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-03-31 23:08, Idan Arye wrote:

 Limiting unittest names to legal identifiers will save a lot of headache
 when we set our tools to actually use these names. Matching a legal
 identifier in a text stream is much easier than matching an arbitrary
 unicode string, even if that string is escaped.
I don't see any problem with arbitrary strings. It works perfectly fine in RSpec, which has much better tooling than D. -- /Jacob Carlborg
Apr 01 2015
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
Sounds like something that should be added as a library rather 
than the core language.
Mar 30 2015
prev sibling next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:
 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei
I agree that using library-defined annotations would be a better approach than language changes. Currently things like tested use the form name("AddPeer") unittest { /* ... */ } Which is nice, because then you can extend it as desired, such as parallel name("AddPeer") unittest The main issue is that at this point, practically every single person has defined: struct Name { string val; } string name(string val) { return Name(val); } Adding common attributes such as this to Phobos or druntime and potentially have the default unittest runner include them would be good.
Mar 30 2015
parent reply Mathias Lang via Digitalmars-d <digitalmars-d puremagic.com> writes:
We do have an ` name` as UDA in Vibe.d, so that'll be a breaking change
(But ` NamedUnittest("name")` will do).
I also think it should be a library solution.

2015-03-31 0:21 GMT+02:00 Kapps via Digitalmars-d <
digitalmars-d puremagic.com>:

 On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:

 I'd like to make a DIP for named unittests. Who can help me with that?


 Andrei
I agree that using library-defined annotations would be a better approach than language changes. Currently things like tested use the form name("AddPeer") unittest { /* ... */ } Which is nice, because then you can extend it as desired, such as parallel name("AddPeer") unittest The main issue is that at this point, practically every single person has defined: struct Name { string val; } string name(string val) { return Name(val); } Adding common attributes such as this to Phobos or druntime and potentially have the default unittest runner include them would be good.
Mar 30 2015
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 30 March 2015 at 23:02:51 UTC, Mathias Lang wrote:
 We do have an ` name` as UDA in Vibe.d, so that'll be a 
 breaking change
 (But ` NamedUnittest("name")` will do).
 I also think it should be a library solution.
I can't afford to care about used names for attributes, same as we still add new symbols to Phobos (though those do break user code for same reason). They can be disambugated using module system. (and yes, this is HUGE drawback of D module system)
Mar 30 2015
prev sibling next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 31/03/2015 10:52 a.m., Andrei Alexandrescu wrote:
 We're having a strong need for named unittests at Facebook for multiple
 reasons.

 1. We have sophisticated tooling that verifies whether unittests are
 flaky. The automated monitor (for e.g. C++) figures whether a given
 unittest fails several times across several commits. Unittests are
 identified by name; relying on file/line is impossible because the line
 of a failure is not stable across changes.

 2. Again for efficient automated testing and flakiness detection, one
 should be able to run only a subset of unittests by mentioning them by
 line in the command line. Note that this implies there's no
 interdependency between distinct unittests, which is fine because the
 new ability is opt-on; I'd say is pure style anyway.

 3. Mentioning unittest names in failure messages helps human
 communication (e.g. "AddPeer is failing after your change"). This is
 impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me with that?


 Andrei
/** * Does something funny * * Uses writeln, to tell the user off. * And perhaps something else too. */ unittest { import std.stdio; import std.ddoc; writeln("BAD USER, now say \"NO\""); assert(readln() == "NO"); assert(ddocParse(__DDOC__).summary == "Does something funny"); } Only one addition required. __DDOC__ to the compiler. The rest is all library solution. Oh and assert should probably be aware of it. void assert(bool value, string text="", string mod = __MODULE__, uint line = __LINE__, string ddoc = __DDOC__); Well along those lines for a prototype.
Mar 30 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-03-30 23:52, Andrei Alexandrescu wrote:
 We're having a strong need for named unittests at Facebook for multiple
 reasons.

 1. We have sophisticated tooling that verifies whether unittests are
 flaky. The automated monitor (for e.g. C++) figures whether a given
 unittest fails several times across several commits. Unittests are
 identified by name; relying on file/line is impossible because the line
 of a failure is not stable across changes.

 2. Again for efficient automated testing and flakiness detection, one
 should be able to run only a subset of unittests by mentioning them by
 line in the command line. Note that this implies there's no
 interdependency between distinct unittests, which is fine because the
 new ability is opt-on; I'd say is pure style anyway.

 3. Mentioning unittest names in failure messages helps human
 communication (e.g. "AddPeer is failing after your change"). This is
 impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me with that?
I completely agree. I always thought that the built-in unit test support wasn't sufficient. All of the above should be possible to implement in library code without needing to change the language. Now, it depends on how far you want to go. We could do something simple as adding a UDA, which has already been suggested by others: name("this is a test") unittest { assert(false); } Personally I would like a complete testing framework like RSpec. I have a very simple implementation [1] of this: unittest { describe("std.uni", { describe("toUpper", { it("converts a string to uppercase", { "foo".toUpper.should.eq("FOO") }); }); describe("toLower", { it("converts a string to lowercase", { "Foo".toLower.should.eq("foo") }); }); }); } With RSpec, which supports multiple formatters, it can look like this: --- Randomized with seed 28149 std.uni toUpper converts a string to uppercase toLower converts a string to lowercase Finished in 0.00078 seconds (files took 0.0931 seconds to load) 2 examples, 0 failures Randomized with seed 28149 --- This shows a failing test: --- Randomized with seed 57730 std.uni toLower converts a string to lowercase toUpper converts a string to uppercase (FAILED - 1) Failures: 1) std.uni toUpper converts a string to uppercase Failure/Error: 'foo'.should == 'FOO' expected: "FOO" got: "foo" (using ==) # ./spec/foo_spec.rb:6:in `block (3 levels) in <top (required)>' Finished in 0.00298 seconds (files took 0.08553 seconds to load) 2 examples, 1 failure Failed examples: rspec ./spec/foo_spec.rb:5 # std.uni toUpper converts a string to uppercase Randomized with seed 57730 --- Or with the TextMate formatter [2]. It shows both passing and failing test. When a test fails to get a link pointing back to the editor and a syntax highlighted snippet of the failing source code. With RSpec it's also possible to specify the --line-number flag which will only run the tests matching a given line number. Or the --example flag which will run all examples (tests) matching the given string. Is any of this interesting to have in Phobos? Otherwise I'll continue working on my own framework. [1] https://github.com/jacob-carlborg/dspec [2] http://thejqr.com/2009/02/06/textmate-rspec-and-dot-spec-party.html -- /Jacob Carlborg
Mar 31 2015
prev sibling next sibling parent "ezneh" <petitv.isat gmail.com> writes:
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:
 We're having a strong need for named unittests at Facebook for 
 multiple reasons.

 1. We have sophisticated tooling that verifies whether 
 unittests are flaky. The automated monitor (for e.g. C++) 
 figures whether a given unittest fails several times across 
 several commits. Unittests are identified by name; relying on 
 file/line is impossible because the line of a failure is not 
 stable across changes.

 2. Again for efficient automated testing and flakiness 
 detection, one should be able to run only a subset of unittests 
 by mentioning them by line in the command line. Note that this 
 implies there's no interdependency between distinct unittests, 
 which is fine because the new ability is opt-on; I'd say is 
 pure style anyway.

 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). 
 This is impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei
Would it be doable to make something like unittest("Say my name") { // tests } and still be backward compatible ? In the other hand, other proposed solutions seem to be good imho.
Mar 31 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Mon, 30 Mar 2015 14:52:36 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 We're having a strong need for named unittests at Facebook for
 multiple reasons.
 
As there have already been suggestions to use UDAs I think we should discuss the fundamental difficulty with unittests: Right now the default implementation works by putting pointers to a test function into ModuleInfo. We could instead add arrays of some 'unittest information' struct to ModuleInfo to support names etc. But we can't make this as extensible and powerful as it should be: In order to support arbitrary UDAs we'd always need some kind of UDA=>runtime serialization. The other option is getting a list of unittests at compile time. (__traits allMEmbers, etc). AFAIK all unittest frameworks supporting UDA use this approach. This is much more powerful and extensible. It might make sense to switch the default implementation. But here's the problem: 1) The compile time approach requires some kind of explicit registration of the unittests. At least one mixin per module. 2) This mixin will usually provide a module constructor. But using module constructors will cause issues with cycle detection. There are some similar usecases with exactly the same issues (benchmark functions, ...). So we'd need a general solution for this. One possible solution for problem 1 would be to have mixins that automatically get mixed in once at module scope in imported modules (a little bit like RTInfo): ----------------------- module benchmark; automodule mixin template Foo() { shared static this() nocycle { //traits allmembers, ... } } ----------------------- module test; import benchmark; //=> automatically inserts mixin Foo; ----------------------- To solve problem problem 2 a nocycle UDA could be used to ignore cycle checking (or depends("benchmark") to only list explicit dependencies).
Mar 31 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
 Am Mon, 30 Mar 2015 14:52:36 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 We're having a strong need for named unittests at Facebook for
 multiple reasons.
 Right now the default implementation works by putting pointers 
 to a
 test function into ModuleInfo. We could instead add arrays of 
 some
 'unittest information' struct to ModuleInfo to support names 
 etc. But
 we can't make this as extensible and powerful as it should be: 
 In order
 to support arbitrary UDAs we'd always need some kind of 
 UDA=>runtime
 serialization.
Most powerful solution would be to simply put attributes for unittest blocks in runtime information for tests (using RTTI it should be possible to define such variadic structure in similar manner as D-style variadic function arguments).
 The other option is getting a list of unittests at compile time.
 (__traits allMEmbers, etc). AFAIK all unittest frameworks
 supporting UDA use this approach. This is much more powerful and
 extensible. It might make sense to switch the default 
 implementation.

 But here's the problem:

 1) The compile time approach requires some kind
    of explicit registration of the unittests. At least one 
 mixin per
    module.
 2) This mixin will usually provide a module constructor. But
    using module constructors will cause issues with cycle 
 detection.
This is not really true. You only need one mixin in the root module(s), rest can be iterated recursively by test runner itself using __traits(allMembers) reflection. Only issue with that approach is that last time I checked there was a DMD bug which prevented from getting complete list of imported modules via allMembers. Should be fixable. And module constructors are not needed at all for this, there is http://dlang.org/traits.html#getUnitTests
Mar 31 2015
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 31 Mar 2015 13:31:58 +0000
schrieb "Dicebot" <public dicebot.lv>:

 But here's the problem:

 1) The compile time approach requires some kind
    of explicit registration of the unittests. At least one 
 mixin per
    module.
 2) This mixin will usually provide a module constructor. But
    using module constructors will cause issues with cycle 
 detection.
This is not really true. You only need one mixin in the root module(s), rest can be iterated recursively by test runner itself using __traits(allMembers) reflection. Only issue with that approach is that last time I checked there was a DMD bug which prevented from getting complete list of imported modules via allMembers. Should be fixable.
But then you still have to explicitly import (or at least name) all modules that should be tested. This is a drawback compared to the current builtin-unittests where you do not have to explicitly import to-be-tested modules. I was thinking of a fully-automated way where you only have to import a module (could even be object-d => no explicit import required) and have all tests run. This can be done by mixing in a module constructor in every module. From that constructor you'd call std.unittest.registerTest(...) and in the main function you could then call std.unittest.runTests(). This way you never need to name or know the tested modules. IIRC std.benchmark used that approach, with the drawback of manual mixins and module constructor cycle issues.
 
 And module constructors are not needed at all for this, there is 
 http://dlang.org/traits.html#getUnitTests
Sure, but I was thinking about a runtime registration scheme as explained above.
Mar 31 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 31 March 2015 at 14:13:29 UTC, Johannes Pfau wrote:
 Am Tue, 31 Mar 2015 13:31:58 +0000
 schrieb "Dicebot" <public dicebot.lv>:

 But here's the problem:

 1) The compile time approach requires some kind
    of explicit registration of the unittests. At least one 
 mixin per
    module.
 2) This mixin will usually provide a module constructor. But
    using module constructors will cause issues with cycle 
 detection.
This is not really true. You only need one mixin in the root module(s), rest can be iterated recursively by test runner itself using __traits(allMembers) reflection. Only issue with that approach is that last time I checked there was a DMD bug which prevented from getting complete list of imported modules via allMembers. Should be fixable.
But then you still have to explicitly import (or at least name) all modules that should be tested. This is a drawback compared to the current builtin-unittests where you do not have to explicitly import to-be-tested modules.
This is true, but importing modules by name can be code that is generated. Which is exactly what I do with dtest [1] and unit-threaded [2]. It's not that big of a deal when it's part of the build system. Atila [1] http://code.dlang.org/my_packages/dtest [2] http://code.dlang.org/my_packages/unit-threaded
 I was thinking of a fully-automated way where you only have to 
 import a
 module (could even be object-d => no explicit import required) 
 and
 have all tests run. This can be done by mixing in a module 
 constructor
 in every module. From that constructor you'd call
 std.unittest.registerTest(...) and in the main function you 
 could then
 call std.unittest.runTests(). This way you never need to name 
 or know
 the tested modules.

 IIRC std.benchmark used that approach, with the drawback of 
 manual
 mixins and module constructor cycle issues.

 
 And module constructors are not needed at all for this, there 
 is http://dlang.org/traits.html#getUnitTests
Sure, but I was thinking about a runtime registration scheme as explained above.
Mar 31 2015
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 03/31/2015 11:01 PM, Atila Neves wrote:
 This is true, but importing modules by name can be code that is
 generated. Which is exactly what I do with dtest [1] and unit-threaded
 [2]. It's not that big of a deal when it's part of the build system.
Dub does it as well (dub test), because if you don't import a module, it might not get linked too.
Mar 31 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 31 Mar 2015 13:31:58 +0000
schrieb "Dicebot" <public dicebot.lv>:

 On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
 Am Mon, 30 Mar 2015 14:52:36 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 We're having a strong need for named unittests at Facebook for
 multiple reasons.
 Right now the default implementation works by putting pointers 
 to a
 test function into ModuleInfo. We could instead add arrays of 
 some
 'unittest information' struct to ModuleInfo to support names 
 etc. But
 we can't make this as extensible and powerful as it should be: 
 In order
 to support arbitrary UDAs we'd always need some kind of 
 UDA=>runtime
 serialization.
Most powerful solution would be to simply put attributes for unittest blocks in runtime information for tests (using RTTI it should be possible to define such variadic structure in similar manner as D-style variadic function arguments).
Yes, one array of TestAttribute[] struct TestAttribute { TypeInfo ti; void* value; } per unittest and an array of unittests in ModuleInfo would likely work. But this works only for unittests and it's still restricted*. I'd prefer a more general solution which also works for benchmarks and similar usecases. *For example you can extract file/line information (useful for IDEs) but you'll have to add explicit fileline UDAs to the tests. With compile-time reflection you can gather this information without additional UDAs.
Mar 31 2015
parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 14:27:45 UTC, Johannes Pfau wrote:
 Am Tue, 31 Mar 2015 13:31:58 +0000
 schrieb "Dicebot" <public dicebot.lv>:

 On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:
 Am Mon, 30 Mar 2015 14:52:36 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 We're having a strong need for named unittests at Facebook 
 for
 multiple reasons.
 Right now the default implementation works by putting 
 pointers to a
 test function into ModuleInfo. We could instead add arrays 
 of some
 'unittest information' struct to ModuleInfo to support names 
 etc. But
 we can't make this as extensible and powerful as it should 
 be: In order
 to support arbitrary UDAs we'd always need some kind of 
 UDA=>runtime
 serialization.
Most powerful solution would be to simply put attributes for unittest blocks in runtime information for tests (using RTTI it should be possible to define such variadic structure in similar manner as D-style variadic function arguments).
Yes, one array of TestAttribute[] struct TestAttribute { TypeInfo ti; void* value; } per unittest and an array of unittests in ModuleInfo would likely work. But this works only for unittests and it's still restricted*. I'd prefer a more general solution which also works for benchmarks and similar usecases. *For example you can extract file/line information (useful for IDEs) but you'll have to add explicit fileline UDAs to the tests. With compile-time reflection you can gather this information without additional UDAs.
Problem with more generic solution is that it is also note demanding and potentially more intrusive - right now UDA don't affect actual types at all. I'd prefer something simple and practical that can work for texts right now - those are special enough case to justify dedicated approach (even if generic rtinfo gets added later)
Mar 31 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-03-31 15:31, Dicebot wrote:

 Most powerful solution would be to simply put attributes for unittest
 blocks in runtime information for tests (using RTTI it should be
 possible to define such variadic structure in similar manner as D-style
 variadic function arguments).
I think the most powerful and most generic solution would be to: 1. Remove the unittest keyword 2. Make code executable at module scope 3. Add support for trailing delegate syntax module foo; unittest("foobar") { } Would be lowered to unittest("foobar", { }); Works with benchmark and other things as well. -- /Jacob Carlborg
Apr 01 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-03-31 11:08, Johannes Pfau wrote:

 But here's the problem:

 1) The compile time approach requires some kind
     of explicit registration of the unittests. At least one mixin per
     module.
 2) This mixin will usually provide a module constructor. But
     using module constructors will cause issues with cycle detection.
When we get RTInfo for modules [1] there shouldn't be any problems. A template will be instantiated once for each module the compiler sees. With that module __traits(getUnitTests) and __traits(allMembers) can be used to access the unit tests, benchmarks, or whatever. Not need for any registration or module constructors. [1] https://github.com/D-Programming-Language/dmd/pull/2271 -- /Jacob Carlborg
Apr 01 2015
prev sibling next sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
I understand the preference to librarize as much as possible, but 
I don't think the desire to sacrifice every possible bit of 
convenience to avoid the tiniest changes to the language is 
always beneficial. I don't say that implementing everything 
inside the compiler is good either though, but in many cases some 
slight changes to the language can make the library solution so 
much more simple and elegant.

In this case, allowing to name a unittest should be a very simple 
language change that'll make any library implementation of the 
rest of the feature more elegant to use, simpler to implement, 
and more consistent with alternative library implementations.


Another argument in favor of language solution - sometimes the 
unittests take the better part of the compilation process, 
especially when the project is a heavily-templated library(Phobos 
is a prime example). When working on a bug discovered by a 
unittest, that means you either have to either build all 
unittests every time you want to check your changes, or copy the 
unittest code a `main` function and work on it there. If unittest 
names are part of the language, it could be possible to instruct 
the compiler to only build a single unittest, which will make the 
iterations much faster.
Mar 31 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:
 I understand the preference to librarize as much as possible, 
 but I don't think the desire to sacrifice every possible bit of 
 convenience to avoid the tiniest changes to the language is 
 always beneficial. I don't say that implementing everything 
 inside the compiler is good either though, but in many cases 
 some slight changes to the language can make the library 
 solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very 
 simple language change that'll make any library implementation 
 of the rest of the feature more elegant to use, simpler to 
 implement, and more consistent with alternative library 
 implementations.
It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient.
Mar 31 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:
 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:
 I understand the preference to librarize as much as possible, 
 but I don't think the desire to sacrifice every possible bit 
 of convenience to avoid the tiniest changes to the language is 
 always beneficial. I don't say that implementing everything 
 inside the compiler is good either though, but in many cases 
 some slight changes to the language can make the library 
 solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very 
 simple language change that'll make any library implementation 
 of the rest of the feature more elegant to use, simpler to 
 implement, and more consistent with alternative library 
 implementations.
It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient.
But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields.
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 7:45 AM, Idan Arye wrote:
 On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:
 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:
 I understand the preference to librarize as much as possible, but I
 don't think the desire to sacrifice every possible bit of convenience
 to avoid the tiniest changes to the language is always beneficial. I
 don't say that implementing everything inside the compiler is good
 either though, but in many cases some slight changes to the language
 can make the library solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very simple
 language change that'll make any library implementation of the rest
 of the feature more elegant to use, simpler to implement, and more
 consistent with alternative library implementations.
It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient.
But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields.
Interesting point! -- Andrei
Mar 31 2015
parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 31/03/2015 20:21, Andrei Alexandrescu a écrit :
 On 3/31/15 7:45 AM, Idan Arye wrote:
 On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:
 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:
 I understand the preference to librarize as much as possible, but I
 don't think the desire to sacrifice every possible bit of convenience
 to avoid the tiniest changes to the language is always beneficial. I
 don't say that implementing everything inside the compiler is good
 either though, but in many cases some slight changes to the language
 can make the library solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very simple
 language change that'll make any library implementation of the rest
 of the feature more elegant to use, simpler to implement, and more
 consistent with alternative library implementations.
It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient.
But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields.
Interesting point! -- Andrei
It will be nice to have named unittest. And better if IDEs will be able to retrieve those names, to allow replay by name, displaying results by names,...
Mar 31 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 21:05:40 UTC, Xavier Bigand wrote:
 Le 31/03/2015 20:21, Andrei Alexandrescu a écrit :
 On 3/31/15 7:45 AM, Idan Arye wrote:
 On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:
 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:
 I understand the preference to librarize as much as 
 possible, but I
 don't think the desire to sacrifice every possible bit of 
 convenience
 to avoid the tiniest changes to the language is always 
 beneficial. I
 don't say that implementing everything inside the compiler 
 is good
 either though, but in many cases some slight changes to the 
 language
 can make the library solution so much more simple and 
 elegant.

 In this case, allowing to name a unittest should be a very 
 simple
 language change that'll make any library implementation of 
 the rest
 of the feature more elegant to use, simpler to implement, 
 and more
 consistent with alternative library implementations.
It isn't simple at all. Name is just one of many meta-values you commonly want to attach to unittest block. Some others: description, dependency, parallelization, benchmark tag, I/O indicator. It is simply impossible to foresee it all in a language feature - but it is exactly kind of data UDA are designed for. All we need is to enhance/fix the language to actually make using of that information convenient.
But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so the only required change is to allow the user to specify that name. This should be much simpler than adding entirely new fields.
Interesting point! -- Andrei
It will be nice to have named unittest. And better if IDEs will be able to retrieve those names, to allow replay by name, displaying results by names,...
Building by unittest name! Imagine - instead of placing temporary code in `main` to develop a new feature or fix a bug, you put in a named unittest and tell your IDE/build-system to only build that unittest(and whatever code needed for it to run). You `writeln` stuff to the console, and when you get some changes to output what you want you change the `writeln`s to `assert`s and proceed to work on the next step. When you are done, all you have to do is tidy it up a bit and BAM - without any special effort you get a unittest that tests that feature/bug you just worked on.
Mar 31 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-31 23:14, Idan Arye wrote:

 Building by unittest name! Imagine - instead of placing temporary code
 in `main` to develop a new feature or fix a bug, you put in a named
 unittest and tell your IDE/build-system to only build that unittest(and
 whatever code needed for it to run). You `writeln` stuff to the console,
 and when you get some changes to output what you want you change the
 `writeln`s to `assert`s and proceed to work on the next step. When you
 are done, all you have to do is tidy it up a bit and BAM - without any
 special effort you get a unittest that tests that feature/bug you just
 worked on.
You just reinvented test driven development ;). It's perfectly possible to do this with a UDA an a text string as well. UDA's also allows you to tag the tests. Basically a short name you put on multiple tests, then tell the test runner to run only those tests, or ignore those. tag("foo") name("this is my test name") unittest {} $ run-tests -t foo -- /Jacob Carlborg
Apr 01 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 1 April 2015 at 14:05:46 UTC, Jacob Carlborg wrote:
 On 2015-03-31 23:14, Idan Arye wrote:

 Building by unittest name! Imagine - instead of placing 
 temporary code
 in `main` to develop a new feature or fix a bug, you put in a 
 named
 unittest and tell your IDE/build-system to only build that 
 unittest(and
 whatever code needed for it to run). You `writeln` stuff to 
 the console,
 and when you get some changes to output what you want you 
 change the
 `writeln`s to `assert`s and proceed to work on the next step. 
 When you
 are done, all you have to do is tidy it up a bit and BAM - 
 without any
 special effort you get a unittest that tests that feature/bug 
 you just
 worked on.
You just reinvented test driven development ;). It's perfectly possible to do this with a UDA an a text string as well. UDA's also allows you to tag the tests. Basically a short name you put on multiple tests, then tell the test runner to run only those tests, or ignore those. tag("foo") name("this is my test name") unittest {} $ run-tests -t foo
The problem is not with running the tests, it's with building them. In heavily templated libraries(like, for example Phobos), building without unittests takes seconds and building with unitetests takes minutes - mainly because the tests need to to many template instantiations. If we could tell the compiler to only build a single, specific test the development cycle can become orders of magnitude faster.
Apr 01 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-04-01 20:35, Idan Arye wrote:

 If we could tell the compiler to only build a single, specific test the
 development cycle can become orders of magnitude faster.
There should be a lot of option to run tests: * Base on a file * Line number * Name * Tag And so on. -- /Jacob Carlborg
Apr 01 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 1 April 2015 at 18:35:14 UTC, Idan Arye wrote:
 The problem is not with running the tests, it's with building 
 them. In heavily templated libraries(like, for example Phobos), 
 building without unittests takes seconds and building with 
 unitetests takes minutes - mainly because the tests need to to 
 many template instantiations. If we could tell the compiler to 
 only build a single, specific test the development cycle can 
 become orders of magnitude faster.
This is fixed by having smaller modules. If test for a single module takes more than few blinks of an eye, something is wrong already.
Apr 01 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-01 21:16, Dicebot wrote:

 This is fixed by having smaller modules. If test for a single module
 takes more than few blinks of an eye, something is wrong already.
For unit tests, yes. But there are other kinds of tests as well. Integration, functional, user acceptance test and so on. -- /Jacob Carlborg
Apr 01 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 2 April 2015 at 06:21:53 UTC, Jacob Carlborg wrote:
 On 2015-04-01 21:16, Dicebot wrote:

 This is fixed by having smaller modules. If test for a single 
 module
 takes more than few blinks of an eye, something is wrong 
 already.
For unit tests, yes. But there are other kinds of tests as well. Integration, functional, user acceptance test and so on.
I don't think anyone is going to put those in a same inline style as unittest blocks, so this is not truly relevant. At least I hope so.
Apr 02 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-04-02 22:31, Dicebot wrote:

 I don't think anyone is going to put those in a same inline style as
 unittest blocks, so this is not truly relevant. At least I hope so.
You mean inline with the code it tests? No, I hope so too. I put my unit tests in separate files as well, but that's just the way I prefer it. -- /Jacob Carlborg
Apr 03 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 14:45:50 UTC, Idan Arye wrote:
 But unittests already have 
 names(http://dpaste.dzfl.pl/b15e94000f15), so the only required 
 change is to allow the user to specify that name. This should 
 be much simpler than adding entirely new fields.
And does not really help. Of all metadata that one may want to attach to a test block, names are actually _least_ important in practice. It will be matter of time until Andrei creates new topic about adding descriptions to tests or way to mark flakey ones. Better to invest into something that scales at least a bit.
Mar 31 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 21:22:09 UTC, Dicebot wrote:
 On Tuesday, 31 March 2015 at 14:45:50 UTC, Idan Arye wrote:
 But unittests already have 
 names(http://dpaste.dzfl.pl/b15e94000f15), so the only 
 required change is to allow the user to specify that name. 
 This should be much simpler than adding entirely new fields.
And does not really help. Of all metadata that one may want to attach to a test block, names are actually _least_ important in practice. It will be matter of time until Andrei creates new topic about adding descriptions to tests or way to mark flakey ones. Better to invest into something that scales at least a bit.
I think you and I work under different assumptions of the goals for this feature. If we only want unittest names to be something that can be printed when the unittest runner runs the unittests, than a UDA with a string is indeed preferable. If we want something that tools can actually use to refer to a specific unittest, we need a proper identifier(yes, even though it can be implemented in library code because D is Turing-complete...)
Mar 31 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-01 00:02, Idan Arye wrote:

 I think you and I work under different assumptions of the goals for this
 feature. If we only want unittest names to be something that can be
 printed when the unittest runner runs the unittests, than a UDA with a
 string is indeed preferable. If we want something that tools can
 actually use to refer to a specific unittest, we need a proper
 identifier(yes, even though it can be implemented in library code
 because D is Turing-complete...)
The tools should, mostly, use the file and line information. -- /Jacob Carlborg
Apr 01 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 1 April 2015 at 14:07:14 UTC, Jacob Carlborg wrote:
 On 2015-04-01 00:02, Idan Arye wrote:

 I think you and I work under different assumptions of the 
 goals for this
 feature. If we only want unittest names to be something that 
 can be
 printed when the unittest runner runs the unittests, than a 
 UDA with a
 string is indeed preferable. If we want something that tools 
 can
 actually use to refer to a specific unittest, we need a proper
 identifier(yes, even though it can be implemented in library 
 code
 because D is Turing-complete...)
The tools should, mostly, use the file and line information.
So now we just need a way to keep a line number unmodified when you add lines above it...
Apr 01 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-04-01 20:49, Idan Arye wrote:

 So now we just need a way to keep a line number unmodified when you add
 lines above it...
I'm not sure what kind of tools you have in mind. * Print a link which points back to a failing test, requires file and line information * Running a specific test, requires file and line information, name or a tag What else do you have in mind? -- /Jacob Carlborg
Apr 01 2015
prev sibling next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 03/31/2015 04:45 PM, Idan Arye wrote:
 
 But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so
 the only required change is to allow the user to specify that name. This
 should be much simpler than adding entirely new fields.
That's the line number, which can't be used as the OP pointed out.
Mar 31 2015
prev sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, March 31, 2015 14:45:49 Idan Arye via Digitalmars-d wrote:
 But unittests already have
 names(http://dpaste.dzfl.pl/b15e94000f15), so the only required
 change is to allow the user to specify that name. This should be
 much simpler than adding entirely new fields.
Exactly what I was thinking. If we just made unittest myTest { } or unittest(myTest) { } legal, then the unit test function would then be named myTest instead of whatever it's already named right now. It's a lot less ugly than using attirbutes, and it fits really well into what we already have. Certainly, it seems like a very straightforward and simple solution to the problem. - Jonathan M Davis
Mar 31 2015
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
I already have a library that finds all unittest blocks (as well 
as its own) and names them by appending a monotonically 
increasing integer to mymodulename.unittest. I've been thinking 
of using a UDA to decorate unittest blocks so that they show up 
with that instead but never got around to it because I don't 
really use unittest blocks and nobody is asking for the feature. 
It'd take 30min to add though.

Obviously 1st class support would be preferable, I like the

unittest nameGoesHere {
    //...
}

that was mentioned later in the thread.

Atila

On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:
 We're having a strong need for named unittests at Facebook for 
 multiple reasons.

 1. We have sophisticated tooling that verifies whether 
 unittests are flaky. The automated monitor (for e.g. C++) 
 figures whether a given unittest fails several times across 
 several commits. Unittests are identified by name; relying on 
 file/line is impossible because the line of a failure is not 
 stable across changes.

 2. Again for efficient automated testing and flakiness 
 detection, one should be able to run only a subset of unittests 
 by mentioning them by line in the command line. Note that this 
 implies there's no interdependency between distinct unittests, 
 which is fine because the new ability is opt-on; I'd say is 
 pure style anyway.

 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). 
 This is impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei
Mar 31 2015
prev sibling next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
 I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use. unittest (myname) vs. unittest ("mynameexpression")
 Andrei
There is an ER and an half of an implementation. https://issues.dlang.org/show_bug.cgi?id=4747 https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 2:50 PM, Martin Nowak wrote:
 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
 I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use. unittest (myname) vs. unittest ("mynameexpression")
Also unittest myname and unittest "mynameexpression". There will be no shortage of folks willing to debate this to smithereens.
 Andrei
There is an ER and an half of an implementation. https://issues.dlang.org/show_bug.cgi?id=4747 https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308
Nice, I'll check that out. Andrei
Mar 31 2015
parent David Gileadi <gileadis NSPMgmail.com> writes:
On 3/31/15 3:28 PM, Andrei Alexandrescu wrote:
 On 3/31/15 2:50 PM, Martin Nowak wrote:
 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
 I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use. unittest (myname) vs. unittest ("mynameexpression")
Also unittest myname and unittest "mynameexpression". There will be no shortage of folks willing to debate this to smithereens.
One argument for myname (using standard identifier naming rules) is that it's command-line friendly: it won't require quoting when invoking a single unit test, and tends toward shorter names. It may also be more obvious that standard identifier names would use the existing lookup rules for guaranteeing uniqueness among unittest names. One argument for "mynameexpression" is that it allows for nice descriptions (e.g. unittest "count should accept a custom comparator function"). One compromise might be something like: description("count should accept a custom comparator function") unittest countCustomComparator Under this example the unittest could be invoked from the command line by the countCustomComparator name or possibly by the description string as an alternative. For the very little it's worth my bikeshed is painted without parentheses, assuming that's easily implemented.
Mar 31 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 31 March 2015 at 21:50:56 UTC, Martin Nowak wrote:
 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
 I'd like to make a DIP for named unittests. Who can help me 
 with that?
Why a DIP, isn't the only question what syntax to use. unittest (myname) vs. unittest ("mynameexpression")
 Andrei
There is an ER and an half of an implementation. https://issues.dlang.org/show_bug.cgi?id=4747 https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308
So now we are going to change the language for this ? There is a natural name for unitests, the name of the module. We have way to break module into pieces in a backward compatible manner now, so it's all good. We may want to add various annotation to a test, and we have UDA for that. So the only things that is really needed is a way to customize the test runner from client code to output whatever everybody needs.
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 3:58 PM, deadalnix wrote:
 On Tuesday, 31 March 2015 at 21:50:56 UTC, Martin Nowak wrote:
 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:
 I'd like to make a DIP for named unittests. Who can help me with that?
Why a DIP, isn't the only question what syntax to use. unittest (myname) vs. unittest ("mynameexpression")
 Andrei
There is an ER and an half of an implementation. https://issues.dlang.org/show_bug.cgi?id=4747 https://github.com/D-Programming-Language/dmd/pull/1131 https://github.com/D-Programming-Language/druntime/pull/308
So now we are going to change the language for this ? There is a natural name for unitests, the name of the module. We have way to break module into pieces in a backward compatible manner now, so it's all good. We may want to add various annotation to a test, and we have UDA for that. So the only things that is really needed is a way to customize the test runner from client code to output whatever everybody needs.
The reasonable course is to see how far we can get with a library-only solution. Amaury, want to work on that? -- Andrei
Mar 31 2015
next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei
In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats
Mar 31 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 4:38 PM, Martin Nowak wrote:
 On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei
In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats
Atila, is there interest in moving your library to std.experimental? -- Andrei
Mar 31 2015
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu 
wrote:
 On 3/31/15 4:38 PM, Martin Nowak wrote:
 On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
 The reasonable course is to see how far we can get with a 
 library-only
 solution. Amaury, want to work on that? -- Andrei
In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats
Atila, is there interest in moving your library to std.experimental? -- Andrei
Sure, I can have the PR ready by next week. As mentioned earlier, a UDA to attach a name to a unittest block and then run with that name in the output is 30min work. Making the whole thing std.experimental ready will take a few days. TBH, if I'd known it had a chance of making it in there, I'd've done it ages ago! Atila
Apr 01 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/15 12:46 AM, Atila Neves wrote:
 On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu wrote:
 On 3/31/15 4:38 PM, Martin Nowak wrote:
 On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:
 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei
In any case you should talk to Atila Neves who wrote a really good unittest library. http://code.dlang.org/packages/unit-threaded http://code.dlang.org/packages/dtest It's also used by quite a lot of people already. http://code.dlang.org/api/packages/unit-threaded/stats
Atila, is there interest in moving your library to std.experimental? -- Andrei
Sure, I can have the PR ready by next week. As mentioned earlier, a UDA to attach a name to a unittest block and then run with that name in the output is 30min work. Making the whole thing std.experimental ready will take a few days. TBH, if I'd known it had a chance of making it in there, I'd've done it ages ago! Atila
Sounds like a plan. Thanks in advance! -- Andrei
Apr 01 2015
prev sibling parent "Robert burner Schadek" <rburners gmail.com> writes:
I have PR 
https://github.com/D-Programming-Language/phobos/pull/2995 open 
since October 2014.

it includes:
* extensible haskell like quickcheck
* benchmarking with "names" and record keeping to see progress
* offline tool to plot benchmark results with gnuplot
* most std.string functions already have benchmarks
* makefiles got a new BUILD=benchmark

IMO continuous benchmarking of phobos is a must-have feature and 
a nice marketing tool ( benchmark graphs on dlang.org). Something 
similar to quickcheck is equally important, to get the test input 
you didn't expect.

I got one and a half comments so far.
Apr 01 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu 
wrote:
 The reasonable course is to see how far we can get with a 
 library-only solution. Amaury, want to work on that? -- Andrei
If nobody picks it up, I can, but I have very little bandwidth right now with the work involved on SDC.
Mar 31 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 5:58 PM, deadalnix wrote:
 On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu wrote:
 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei
If nobody picks it up, I can, but I have very little bandwidth right now with the work involved on SDC.
I'll quote that when you post a long rant here :o). -- Andrei
Mar 31 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/01/2015 12:58 AM, deadalnix wrote:
 So now we are going to change the language for this ?
 
 There is a natural name for unitests, the name of the module. We have
 way to break module into pieces in a backward compatible manner now, so
 it's all good.
Are you saying one should split off unittests into submodules? Or are you saying https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d is enough for your needs? I think having multiple tests per module makes sense, e.g. because it's better to parallelize the tests.
 We may want to add various annotation to a test, and we have UDA for that.
Sure use a UDA, I don't care. It seems important though, that the compiler recognizes it and emits it as metadata to the ModuleInfo. And if you need some compiler magic anyhow, a UDA just looks redundant. unittest("my test") unittest unittest("my test")
 So the only things that is really needed is a way to customize the test
 runner from client code to output whatever everybody needs.
You mean this? http://dlang.org/phobos/core_runtime.html#.Runtime.moduleUnitTester
Mar 31 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 31 March 2015 at 23:29:28 UTC, Martin Nowak wrote:
 On 04/01/2015 12:58 AM, deadalnix wrote:
 So now we are going to change the language for this ?
 
 There is a natural name for unitests, the name of the module. 
 We have
 way to break module into pieces in a backward compatible 
 manner now, so
 it's all good.
Are you saying one should split off unittests into submodules?
No necessarily. Typically, in most test framework, you have a class, and various method, each of them testing various aspect of what you want to test. I see the class here as the module, and each of the method as a unitest block.
 Or are you saying
 https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d
 is enough for your needs?

 I think having multiple tests per module makes sense, e.g. 
 because it's
 better to parallelize the tests.
See my previous comment, that is not needed. In fact i also think having several unitest blocks is useful.
 We may want to add various annotation to a test, and we have 
 UDA for that.
Sure use a UDA, I don't care. It seems important though, that the compiler recognizes it and emits it as metadata to the ModuleInfo. And if you need some compiler magic anyhow, a UDA just looks redundant.
How I see it, it should be possible for a user to specify a test runner template. This test runner template can do compile time reflection and format its output the way to wants. We must not bake all required attribute we need to associate a test in the language. What if we want to associate the test with a team name, or give it a name, or give an id that my tools uses to check for tests, or whatever. Every user will want to add various metadata, and these will vary. Every users will want various output, and these will vary. It should be possible to have a test runner that use template and compile time reflection to make sense of the unitest block. Yes, there is a bit of compiler magic involved, but nothing terribly fancy.
 You mean this?
 http://dlang.org/phobos/core_runtime.html#.Runtime.moduleUnitTester
We need template to use compile time reflection.
Mar 31 2015
parent "Panke" <tobias pankrath.net> writes:
 We need template to use compile time reflection.
Please don't make it all compile time. It is useful to register tests at runtime as well. Like one tests per SAT instance in a given SAT benchmark. Practically only thing we need is a standard way to register a test function as a unittest and associate a key-value store with it.
Mar 31 2015
prev sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2015-03-30 21:52:36 +0000, Andrei Alexandrescu said:

 We're having a strong need for named unittests at Facebook for multiple 
 reasons.
 
 1. We have sophisticated tooling that verifies whether unittests are 
 flaky. The automated monitor (for e.g. C++) figures whether a given 
 unittest fails several times across several commits. Unittests are 
 identified by name; relying on file/line is impossible because the line 
 of a failure is not stable across changes.
 
 2. Again for efficient automated testing and flakiness detection, one 
 should be able to run only a subset of unittests by mentioning them by 
 line in the command line. Note that this implies there's no 
 interdependency between distinct unittests, which is fine because the 
 new ability is opt-on; I'd say is pure style anyway.
 
 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). This is 
 impossible with file and line numbers.
 
 I'd like to make a DIP for named unittests. Who can help me with that?
 
 
 Andrei
This is a fantastic idea -- and something I was concerned about lately. +1 I can possibly help with a DIP, but it seems like a very straightforward request? -Shammah
Mar 31 2015
parent "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 1 April 2015 at 04:42:33 UTC, Shammah Chancellor 
wrote:
 I can possibly help with a DIP, but it seems like a very 
 straightforward request?

 -Shammah
Considering the heated debate going on here, I'd say it's not as straightforward as it seems...
Apr 01 2015