www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Let's bikeshed std.experimental.testing

reply "Atila Neves" <atila.neves gmail.com> writes:
In case you don't know what I'm talking about: 
https://github.com/D-Programming-Language/phobos/pull/3207

Since this is an API issue it's import to get it right the first 
time. Personally I'm not sure what I prefer (well, I am, but what 
I actually want isn't syntactically valid D). I think the options 
so far are:

1) What's there already, namely `shouldEquals`, `shouldBeIn`, etc.
2a) Compile-time strings for operators: `should!"=="`, 
`should!"in"`
2b) Dicebot's `test!"=="`. `assert` is so much better, I wish we 
could use that.
3) Composable ones: should.equals, should.not.equals, or another 
word that isn't "should"
4) Anything else?

I'm not convinced composability brings anything to the table 
except for editor dot-completion. I don't like the verbosity of 
what's there now, but my prefererred syntax doesn't work except 
for the ubiquitous  check for equality (`should ==`). Well, the 
dream would be that `assert(foo == bar)` did what part of this PR 
does, but that's another story and something that can't be done 
by a library unless we had AST macros, which we won't. Or Lisp's 
reader macros, but we won't get those either.

Thoughts? Votes?

Atila
Jun 30 2015
next sibling parent reply "Adrian Matoga" <epi atari8.info> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`).
Could you give some examples of your preferred syntax and why it doesn't work?
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 08:38:44 UTC, Adrian Matoga wrote:
 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity 
 of what's there now, but my prefererred syntax doesn't work 
 except for the ubiquitous  check for equality (`should ==`).
Could you give some examples of your preferred syntax and why it doesn't work?
`foo.should == "bar";` works. Nothing else does (and in fact in retrospect it's surprising that == does) because they do nothing by themselves. `foo.should != bar` is the same as `!(foo == bar)`, which on a statement by itself is nonsensical and rejected by the compiler with "has no effect in expression" error. You could write something like `if(foo.should != "bar") {}` and that compiles fine but it's super hacky and ugly. Atila
Jun 30 2015
parent "Adrian Matoga" <epi atari8.info> writes:
On Tuesday, 30 June 2015 at 11:14:55 UTC, Atila Neves wrote:
 On Tuesday, 30 June 2015 at 08:38:44 UTC, Adrian Matoga wrote:
 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity 
 of what's there now, but my prefererred syntax doesn't work 
 except for the ubiquitous  check for equality (`should ==`).
Could you give some examples of your preferred syntax and why it doesn't work?
`foo.should == "bar";` works. Nothing else does (and in fact in retrospect it's surprising that == does) because they do nothing by themselves. `foo.should != bar` is the same as `!(foo == bar)`, which on a statement by itself is nonsensical and rejected by the compiler with "has no effect in expression" error. You could write something like `if(foo.should != "bar") {}` and that compiles fine but it's super hacky and ugly. Atila
Thanks. I took a second look and now it looks obvious. I'd vote for Dicebot's bikeshed design (2b). It's short and looks equally readable for every possible test. Another hackish possibility could be to change the behavior of assert so that it also passes the stringified expression to _d_assert* and _d_unittest* in druntime.
Jun 30 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
To do it _really_ nice, we would need some sort of ".codeof" 
feature. Possibly, with implicit returns in lambdas. For example:

void test(alias expr)
{
     if (!expr())
         throw new TestException(
             expr.codeof ~ " has failed,\n" ~
             /* investigate expr context pointer and grab a/b */
             "\ta = x\n" ~
             "\tb = y"
}

unittest
{
     int a, b;
     test!({ a == b; });
}

Of course, this is one of cases where AST macros would really 
shine. But I think it should be possible to provide necessary 
functionality subset in a much more simple and limited feature.

In absence of language changes, I don't see anything as clear and 
simple as operator mixins. Less magic in unittests -> better. 
Common misconception IMHO is that tests should look nice for 
library/app author, while, from the ecosystem PoV, they should 
look simple and predictable for contributors - and that is most 
important property.
Jun 30 2015
parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 30 June 2015 at 11:43:36 UTC, Dicebot wrote:
 In absence of language changes, I don't see anything as clear 
 and simple as operator mixins. Less magic in unittests -> 
 better. Common misconception IMHO is that tests should look 
 nice for library/app author, while, from the ecosystem PoV, 
 they should look simple and predictable for contributors - and 
 that is most important property.
I tend to agree with your position on testing frameworks. It seems really cool to utilize English to spell out an expectation, but it ends up more complicated. For one, it can never actually be English (I wouldn't want it to be) and for another, I've already got expressions from the language being used that mean the same thing. unittest { struct A { int m; } A a; A b; a.m = 5; b.m = 5; auto testA = a.test; with(testA) { verify(testA.lhs == b); b.m = 6; verify(testA.lhs != b); verify(testA.lhs == b); } } It is kind of like what you're saying, but the condition isn't reportable. I'm actually less concerned about the condition as I am being able to report all the values utilized by the condition. I'd like to see assert be more descriptive or a framework that looks like assert and is more descriptive. Here is the crap code to run the above: https://gist.github.com/JesseKPhillips/df79479cbf6a0e3c6b0d
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 1 July 2015 at 00:26:21 UTC, Jesse Phillips wrote:
 On Tuesday, 30 June 2015 at 11:43:36 UTC, Dicebot wrote:
 [...]
I tend to agree with your position on testing frameworks. It seems really cool to utilize English to spell out an expectation, but it ends up more complicated. For one, it can never actually be English (I wouldn't want it to be) and for another, I've already got expressions from the language being used that mean the same thing. [...]
opEquals is easy. It's everything else that's hard. Atila
Jul 01 2015
parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Wednesday, 1 July 2015 at 08:22:50 UTC, Atila Neves wrote:
 opEquals is easy. It's everything else that's hard.

 Atila
I thought that opNotEquals was also a challenge. The way I approached it should be usable by all operator overloads, but I'm not making the claim that the way I approached it was any good. Only saying its the direction I prefer.
Jul 01 2015
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 - snip -

 Thoughts? Votes?
I really hate the 3rd persons "s" in "shouldEquals" / "should.equals". Not only is it grammatically wrong, it's also inconsistent with "shouldBeIn", which - following the same scheme - would have to be "shouldIsIn".
Jun 30 2015
prev sibling next sibling parent "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 In case you don't know what I'm talking about: 
 https://github.com/D-Programming-Language/phobos/pull/3207

 Since this is an API issue it's import to get it right the 
 first time. Personally I'm not sure what I prefer (well, I am, 
 but what I actually want isn't syntactically valid D). I think 
 the options so far are:

 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`). Well, the 
 dream would be that `assert(foo == bar)` did what part of this 
 PR does, but that's another story and something that can't be 
 done by a library unless we had AST macros, which we won't. Or 
 Lisp's reader macros, but we won't get those either.

 Thoughts? Votes?

 Atila
Google uses gMock. On top of being a mock framework it provides composable matchers API which are a great way of expressing what you want to test. https://www.youtube.com/watch?v=sYpCyLI47rM?t=19m15s (excerpt from outdated gMock presentation from 2008) The matcher api is based on Hamcrest and I think it's pretty convenient. Crash or log : assertThat(...); expectThat(...); Simple tests are easy to express. assertThat(4, 5); More complex things: assertThat(theBiscuit, is(equalTo(myBiscuit))); assertThat(theBiscuit, not(instanceOf(Liquid.class))); Same thing with container: assertThat(map, not(hasKey(lessThan(5)))); The tests are pretty readable and predicates compose nicely. Also because you express the condition as a tree of predicates the error reporting can be really nice. Like "map contains element which key is not less than 5". Also one can write custom predicates for more specific tests. The C++ implementation relies on ugly template/macros. I'm sure a D implementation can be cleaner and pretty sweet to use. I suspect throwing ufcs in here will makes it even better.
Jun 30 2015
prev sibling next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
I think these are both bad, for the reason that ! also means logical not. I read `should!"=="` as "should not equal" before catching myself.
Jun 30 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 12:00:36 UTC, Meta wrote:
 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
I think these are both bad, for the reason that ! also means logical not. I read `should!"=="` as "should not equal" before catching myself.
I'd forgotten about it, but I knew there was a reason I didn't like `should!"=="`, and it was the same reason as you. Atila
Jun 30 2015
prev sibling next sibling parent reply "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 In case you don't know what I'm talking about: 
 https://github.com/D-Programming-Language/phobos/pull/3207

 Since this is an API issue it's import to get it right the 
 first time. Personally I'm not sure what I prefer (well, I am, 
 but what I actually want isn't syntactically valid D). I think 
 the options so far are:

 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`). Well, the 
 dream would be that `assert(foo == bar)` did what part of this 
 PR does, but that's another story and something that can't be 
 done by a library unless we had AST macros, which we won't. Or 
 Lisp's reader macros, but we won't get those either.

 Thoughts? Votes?

 Atila
Much rather prefer the composable ones over the `shouldEquals`, simply for readability and easy extending. These days I am leaning towards BDD, but everybody has his favorite. Maybe just providing the low-level details in std.testing would enough; e.g. a test runner, UDA's and assertions. Then everyone can write his on version of given().when().then() on top of it. Or simply make a pull-request for std.testing.bdd
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 12:42:40 UTC, Sebastiaan Koppe wrote:
 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 [...]
Much rather prefer the composable ones over the `shouldEquals`, simply for readability and easy extending. These days I am leaning towards BDD, but everybody has his favorite. Maybe just providing the low-level details in std.testing would enough; e.g. a test runner, UDA's and assertions. Then everyone can write his on version of given().when().then() on top of it. Or simply make a pull-request for std.testing.bdd
Yeah, I'm starting to think it might be better to delete `should.d` from my current PR, try to get the rest approved then work on where the community wants the fancy assertions to go. It's a shame though because I think it's a massively important piece of the whole thing. It's a night and day difference when a test fails. Atila
Jun 30 2015
next sibling parent "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Tuesday, 30 June 2015 at 14:58:45 UTC, Atila Neves wrote:
 On Tuesday, 30 June 2015 at 12:42:40 UTC, Sebastiaan Koppe 
 wrote:
 On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 [...]
These days I am leaning towards BDD, but everybody has his favorite. Maybe just providing the low-level details in std.testing would enough; e.g. a test runner, UDA's and assertions.
Yeah, I'm starting to think it might be better to delete `should.d` from my current PR, try to get the rest approved then work on where the community wants the fancy assertions to go. It's a shame though because I think it's a massively important piece of the whole thing. It's a night and day difference when a test fails. Atila
Makes sense. You could still keep the should's, just rename the whole lot to isEmpty / isNotEmpty / isGreaterThan and have it return a bool instead of calling fail internally. Then you would simply expect the callee to do that. As in: `assert(5.isEqual(6));`.
Jun 30 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 30/06/15 16:58, Atila Neves wrote:

 Yeah, I'm starting to think it might be better to delete `should.d` from
 my current PR, try to get the rest approved then work on where the
 community wants the fancy assertions to go. It's a shame though because
 I think it's a massively important piece of the whole thing. It's a
 night and day difference when a test fails.
Yeah, I agree, it would be a shame to not have these assertions. -- /Jacob Carlborg
Jun 30 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 30/06/15 10:06, Atila Neves wrote:

 Well, the dream would be that `assert(foo ==
 bar)` did what part of this PR does, but that's another story and
 something that can't be done by a library unless we had AST macros,
 which we won't. Or Lisp's reader macros, but we won't get those either.
I was thinking the same. Both the test!"==" and shouldEqual are workarounds to get a nice message on an assertion failure. I'm wondering how hard it would be to have the compiler generate a string representing the failing expression. -- /Jacob Carlborg
Jun 30 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 19:48:46 UTC, Jacob Carlborg wrote:
 On 30/06/15 10:06, Atila Neves wrote:

 Well, the dream would be that `assert(foo ==
 bar)` did what part of this PR does, but that's another story 
 and
 something that can't be done by a library unless we had AST 
 macros,
 which we won't. Or Lisp's reader macros, but we won't get 
 those either.
I was thinking the same. Both the test!"==" and shouldEqual are workarounds to get a nice message on an assertion failure. I'm wondering how hard it would be to have the compiler generate a string representing the failing expression.
Well, there's a PR for improving assertions here: https://github.com/D-Programming-Language/dmd/pull/1426 Since I have 0 experience in compilers in general and dmd in particular, I thought that'd be an easy way for me to get an in on the assert situation. It seems... more complicated than I can handle at the moment. The 100% ideal situation is for assert to do what I'm doing with the functions in the `should` module. That module really shouldn't even exist. Atila
Jul 01 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 1 July 2015 at 08:21:28 UTC, Atila Neves wrote:
 Well, there's a PR for improving assertions here: 
 https://github.com/D-Programming-Language/dmd/pull/1426
Sadly, this does not really fix the issue in the long term. Using assertions in tests prevents building non-fatal test runners because by spec AssertError is non-recoverable. That is quite a serious limitation. To really fix it, this information needs to be available in library code.
Jul 01 2015
parent Jacob Carlborg <doob me.com> writes:
On 01/07/15 15:45, Dicebot wrote:

 Sadly, this does not really fix the issue in the long term. Using
 assertions in tests prevents building non-fatal test runners because by
 spec AssertError is non-recoverable. That is quite a serious limitation.
There's an assertion handler that can be set in druntime [1]. Unfortunately the callback is declared as "nothrow". If this "nothrow" it could be used to throw an exception instead of an error. [1] https://github.com/D-Programming-Language/druntime/blob/master/src/core/exception.d#L378 -- /Jacob Carlborg
Jul 01 2015
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
https://www.youtube.com/watch?v=pWS8Mg-JWSg&feature=player_detailpage#t=81
Jun 30 2015
prev sibling next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 30 June 2015 at 08:06:37 UTC, Atila Neves wrote:
 In case you don't know what I'm talking about: 
 https://github.com/D-Programming-Language/phobos/pull/3207

 Since this is an API issue it's import to get it right the 
 first time. Personally I'm not sure what I prefer (well, I am, 
 but what I actually want isn't syntactically valid D). I think 
 the options so far are:

 1) What's there already, namely `shouldEquals`, `shouldBeIn`, 
 etc.
 2a) Compile-time strings for operators: `should!"=="`, 
 `should!"in"`
 2b) Dicebot's `test!"=="`. `assert` is so much better, I wish 
 we could use that.
 3) Composable ones: should.equals, should.not.equals, or 
 another word that isn't "should"
 4) Anything else?

 I'm not convinced composability brings anything to the table 
 except for editor dot-completion. I don't like the verbosity of 
 what's there now, but my prefererred syntax doesn't work except 
 for the ubiquitous  check for equality (`should ==`). Well, the 
 dream would be that `assert(foo == bar)` did what part of this 
 PR does, but that's another story and something that can't be 
 done by a library unless we had AST macros, which we won't. Or 
 Lisp's reader macros, but we won't get those either.

 Thoughts? Votes?

 Atila
After much thinking and some _incredibly_ hacky attempts to come up with a library solution that can use operators (it was really hacky, I was using `assert()` and making the expression always return true), I came to the conclusion that what I have there already is probably as good as it's going to get bar language changes. The reason is this: D has very smartly side-stepped C++'s problems with operator overloading and boilerplate by making things like `a >=b` be translated to `a.opCmp(b) >=0`. This is one of them Good Things. The same with `!=`, it's `!(a.opEquals(b)`. Again, good. But for test asserts, that means that by the time any possible library code gets to handle it there's been a loss of information. There's so way to know whether opEquals was called with or without negation, or if opCmp was called because the user typed `>`, `>=`, `<` or `<=`. I can make this work: 3.timesTwo.should == 6; 3.timesTwo.should.not == 5; But anything with opCmp is a no-go. The only other operator that would make sense (to me) to overload is opBinary!("in"), but the naming there is a problem. `should.in`? `should.beIn`? Oh wait, that's not the operator anymore. Which brings me to... Naming. Any one word that isn't `assert` is... not as good. It always breaks down for one use case or another, and I think what's in the PR is the best I've heard so far on this forum or in the comments. A compile-time string is... ugly and error-prone especially since the most common assertion is for equality and `!"=="` looks too much like `!=`. So, despite the fact that I wrote `shouldBeGreaterThan`, I hate its verbosity and all I really want to write is `assert(a > b)` and get a decent error message. I don't buy that `should.` is more extensible. For two reasons, first because in all the test frameworks I've seen there aren't that many more types of assertions, and secondly because adding a member function to the struct returned by `should` and adding a new `shouldCamelCase` function is the same amount of work. As for auto-completion, I also see no real difference between the two. Other than assert being semi-magical, I don't see how it can be any better. Atila
Jul 01 2015
next sibling parent reply "Mike" <none none.com> writes:
On Wednesday, 1 July 2015 at 08:40:24 UTC, Atila Neves wrote:
 I hate its verbosity and all I really want to write is 
 `assert(a > b)` and get a decent error message.
I haven't been following this too closely as there is only so much I can pay attention to at any time, so I apologize in advance if the following comments are uninformed. If so, just ignore them. I suspect from your comments that you want to use `assert`, but it's currently reserved by the language and https://github.com/D-Programming-Language/dmd/pull/1426 was not followed through on, and is currently owned by a ghost (literally). I therefore suggest using `assertThat(whatever)`. If/when `assert` is improved upon to allow integration into this library, `assertThat` can be deprecated. Sorry if I'm making noise, Mike
Jul 01 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 1 July 2015 at 09:08:22 UTC, Mike wrote:
 On Wednesday, 1 July 2015 at 08:40:24 UTC, Atila Neves wrote:
 I hate its verbosity and all I really want to write is 
 `assert(a > b)` and get a decent error message.
I haven't been following this too closely as there is only so much I can pay attention to at any time, so I apologize in advance if the following comments are uninformed. If so, just ignore them. I suspect from your comments that you want to use `assert`, but it's currently reserved by the language and https://github.com/D-Programming-Language/dmd/pull/1426 was not followed through on, and is currently owned by a ghost (literally). I therefore suggest using `assertThat(whatever)`. If/when `assert` is improved upon to allow integration into this library, `assertThat` can be deprecated.
The problem there is `assertThat` gives no more information than `assert` currently does. This thread is all about how do we want to deal with custom assertions that give you useful information. My conclusion so far has been that what's there is probably the best we're going to get. I could always change the shouldXXX functions to be called assertXXX instead, but it doesn't look as good with UFCS. That might get me Dicebot's vote though :) Atila Atila
 Sorry if I'm making noise,
 Mike
Jul 01 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 01/07/15 10:40, Atila Neves wrote:

 So, despite the fact that I wrote `shouldBeGreaterThan`, I hate its
 verbosity
You could write "shouldBe.gt(value)".
 I don't buy that `should.` is more extensible. For two reasons, first
 because in all the test frameworks I've seen there aren't that many more
 types of assertions
In every project I have used RSpec I have added custom matchers/assertions. Just a couple of days ago I added a custom matcher to one of my projects: code = code_to_file('void foo() {}') code.should be_parsed_as('meta.definition.method.d') The point of these custom matchers/assertions are that they should make the tests (or specs in the BDD case) more readable and provide better assertion failure messages. Of course, none of these assertions are necessary, we could all just use "assert" or "shouldEqual", in the end every assertion is just a boolean condition. My test could instead look like this, without using a custom matcher: file = code_to_file('void foo() {}') result = `spec/bin/gtm < "#{file.path}" Syntaxes/D.tmLanguage 2>&1` line = result.split("\n").first scope = 'meta.definition.method.d' assert line =~ /#{Regexp.escape(scope)}/ Which one of the two versions are more readable?
 and secondly because adding a member function to
 the struct returned by `should` and adding a new `shouldCamelCase`
 function is the same amount of work.
The idea would be that the part after "should" would be a free function and use UFCS to call it. Should should() { return Should(); } void beCamelCase(Should s); should.beCamelCase That's what allows to create custom assertions. A user of the library can't extend the struct returned by "should". -- /Jacob Carlborg
Jul 01 2015
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:
 On 01/07/15 10:40, Atila Neves wrote:

 [...]
You could write "shouldBe.gt(value)".
 [...]
In every project I have used RSpec I have added custom matchers/assertions. Just a couple of days ago I added a custom matcher to one of my projects: [...]
Ah, makes sense. I think I'm convinced now. The UFCS as an extension mechanism could indeed be handy. Atila
Jul 02 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Thursday, 2 July 2015 at 12:22:31 UTC, Atila Neves wrote:
 On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:
 On 01/07/15 10:40, Atila Neves wrote:

 [...]
You could write "shouldBe.gt(value)".
 [...]
In every project I have used RSpec I have added custom matchers/assertions. Just a couple of days ago I added a custom matcher to one of my projects: [...]
Ah, makes sense. I think I'm convinced now. The UFCS as an extension mechanism could indeed be handy. Atila
So... unconvinced again. I tried implementing it and it started getting to be a right royal pain, and then I realised that there's nothing to prevent a user from writing their own matchers right now. From your example: void shouldBeParsedAs(Code code, ASTNode node) { //I don't really know what the types should be if(...) { throw new UnitTestException(...); } } And... done. No need for a `Should` struct, no need for the complications I faced trying to use one. I think the design is as good as it can be. Atila
Jul 10 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-07-10 14:37, Atila Neves wrote:

 So... unconvinced again. I tried implementing it and it started getting
 to be a right royal pain, and then I realised that there's nothing to
 prevent a user from writing their own matchers right now. From your
 example:

      void shouldBeParsedAs(Code code, ASTNode node) {  //I don't really
 know what the types should be
          if(...) {
              throw new UnitTestException(...);
          }
      }

 And... done. No need for a `Should` struct, no need for the
 complications I faced trying to use one. I think the design is as good
 as it can be.
If I compare this with RSpec. A custom matcher in RSpec would implement one method, returning a bool, to indicate if the assertion passed or not. In your example the user need to worry about throwing exceptions and which kind to throw. I consider that an implementation detail. Also, RSpec is able to automatically infer a failure message based on the name of the matcher. Optionally the matcher can customize the failing message. It seems like with your version you need to implement both a positive and negative version. -- /Jacob Carlborg
Jul 12 2015
parent "Atila Neves" <atila.neves gmail.com> writes:
On Sunday, 12 July 2015 at 15:15:07 UTC, Jacob Carlborg wrote:
 On 2015-07-10 14:37, Atila Neves wrote:

 So... unconvinced again. I tried implementing it and it 
 started getting
 to be a right royal pain, and then I realised that there's 
 nothing to
 prevent a user from writing their own matchers right now. From 
 your
 example:

      void shouldBeParsedAs(Code code, ASTNode node) {  //I 
 don't really
 know what the types should be
          if(...) {
              throw new UnitTestException(...);
          }
      }

 And... done. No need for a `Should` struct, no need for the
 complications I faced trying to use one. I think the design is 
 as good
 as it can be.
If I compare this with RSpec. A custom matcher in RSpec would implement one method, returning a bool, to indicate if the assertion passed or not. In your example the user need to worry about throwing exceptions and which kind to throw. I consider that an implementation detail. Also, RSpec is able to automatically infer a failure message based on the name of the matcher. Optionally the matcher can customize the failing message. It seems like with your version you need to implement both a positive and negative version.
There's a reason for that and I'm curious as to how RSpec handles it. If it were just a matter of pairing up assertions for the two boolean values of a condition that'd be easy. But the formatting of the error message is anything but. `shouldEqual` and `shouldNotEqual` have very different outputs even if what causes an exception to be thrown is just a check on the opposite condition. For the same reason, a user wanting to extend the framework has to format their own error messages, which means just defining a predicate isn't enough. If I'd seen any more duplication I'd've removed it, but the different positive and negative versions of the `should` assertions are due to the very asymmetrical error messages. If there's a better way of doing it I'd love to find out. Atila
Jul 13 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:
 In every project I have used RSpec I have added custom 
 matchers/assertions. Just a couple of days ago I added a custom 
 matcher to one of my projects:

 code = code_to_file('void foo() {}')
 code.should be_parsed_as('meta.definition.method.d')

 The point of these custom matchers/assertions are that they 
 should make the tests (or specs in the BDD case) more readable 
 and provide better assertion failure messages. Of course, none 
 of these assertions are necessary, we could all just use 
 "assert" or "shouldEqual", in the end every assertion is just a 
 boolean condition.

 My test could instead look like this, without using a custom 
 matcher:

 file = code_to_file('void foo() {}')
 result = `spec/bin/gtm < "#{file.path}" Syntaxes/D.tmLanguage 
 2>&1`
 line = result.split("\n").first
 scope = 'meta.definition.method.d'
 assert line =~ /#{Regexp.escape(scope)}/

 Which one of the two versions are more readable?
Neither. But with the second one I at least have a chance to figure out what it actually tests. To understand the first one I'd need to read the code of that custom matcher. This is exactly the problem with all that fancy - library/app author (who by definition knows the domain well) implicitly introduces custom DSL and expects everyone (who, by definition, have very vague understanding of domain) to be able to read it as comfortably as himself.
Jul 02 2015
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 02/07/15 14:28, Dicebot wrote:

 Neither. But with the second one I at least have a chance to figure out
 what it actually tests. To understand the first one I'd need to read the
 code of that custom matcher.
Once you know what a matcher does or how it's used it's a lot more readable and easier to create new tests.
 This is exactly the problem with all that fancy - library/app author
 (who by definition knows the domain well) implicitly introduces custom
 DSL and expects everyone (who, by definition, have very vague
 understanding of domain) to be able to read it as comfortably as himself.
By that definition we should all write object oriented code in C. Heck, why don't we just use assembly and be done with it. -- /Jacob Carlborg
Jul 02 2015
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 2 July 2015 at 19:06:22 UTC, Jacob Carlborg wrote:
 On 02/07/15 14:28, Dicebot wrote:

 Neither. But with the second one I at least have a chance to 
 figure out
 what it actually tests. To understand the first one I'd need 
 to read the
 code of that custom matcher.
Once you know what a matcher does or how it's used it's a lot more readable and easier to create new tests.
I am not going to investigate all custom matchers and stuff to submit one simple pull request to a project I don't care much about. Simple helper function may be slightly less "pretty" but much easier to grep for and reason about.
 By that definition we should all write object oriented code in 
 C. Heck, why don't we just use assembly and be done with it.
That is exactly why language features are better than free form AST macros. Any abstraction has inherent learning costs. Any non-standard abstraction - even more so. For in-house project it tends to be still worth it because maintenance costs tend to be higher than learning costs and development team is relatively stable. For open-source ecosystem - quite the contrary. And here we speak about something that lays foundation to any open-source contributions - standard testing system.
Jul 02 2015
prev sibling parent "Sebastiaan Koppe" <mail skoppe.eu> writes:
On Thursday, 2 July 2015 at 12:28:41 UTC, Dicebot wrote:
 On Wednesday, 1 July 2015 at 19:38:20 UTC, Jacob Carlborg wrote:
 In every project I have used RSpec I have added custom 
 matchers/assertions. Just a couple of days ago I added a 
 custom matcher to one of my projects:

 code = code_to_file('void foo() {}')
 code.should be_parsed_as('meta.definition.method.d')

 The point of these custom matchers/assertions are that they 
 should make the tests (or specs in the BDD case) more readable 
 and provide better assertion failure messages. Of course, none 
 of these assertions are necessary, we could all just use 
 "assert" or "shouldEqual", in the end every assertion is just 
 a boolean condition.

 My test could instead look like this, without using a custom 
 matcher:

 file = code_to_file('void foo() {}')
 result = `spec/bin/gtm < "#{file.path}" Syntaxes/D.tmLanguage 
 2>&1`
 line = result.split("\n").first
 scope = 'meta.definition.method.d'
 assert line =~ /#{Regexp.escape(scope)}/

 Which one of the two versions are more readable?
Neither. But with the second one I at least have a chance to figure out what it actually tests. To understand the first one I'd need to read the code of that custom matcher.
The good thing about the first one is that you give it a name. The names tells you something about what it does, without having to understand the low-level details - but also, it gives you something to refer to in the documentation. Moreover, it is the inevitable outcome after one has copy-and-pasted the second version 10 times.
 This is exactly the problem with all that fancy - library/app 
 author (who by definition knows the domain well) implicitly 
 introduces custom DSL and expects everyone (who, by definition, 
 have very vague understanding of domain) to be able to read it 
 as comfortably as himself.
A natural consequence of having a domain in the first place.
Jul 03 2015
prev sibling parent reply "linkrope" <linkrope github.com> writes:
1. 'assert' is the wrong thing

I do not need a stack trace for a failed expectation in a unit 
test.
But, usually, I need a stack trace for a contract violation from 
deep down in the unit under test:

https://github.com/linkrope/dunit#failures-vs-errors

Consequently, the failed expectation should throw some "failed 
expectation" 'Exception' instead of an 'AssertError'. So, 
std.exception.assertThrown cannot be used for expectations.

2. something like an 'assert' with better diagnostics could be 
nice

Catch for C++ introduced 'REQUIRE' with helpful failure messages:

     Example.cpp:9: FAILED:
       REQUIRE( Factorial(0) == 1 )
     with expansion:
       0 == 1

https://github.com/philsquared/Catch

3. 'should' seems to be obsolete

Now, 'expect' should be used for expectations:

http://rspec.info/blog/2012/06/rspecs-new-expectation-syntax/
Jul 03 2015
parent Jacob Carlborg <doob me.com> writes:
On 03/07/15 16:07, linkrope wrote:

 3. 'should' seems to be obsolete

 Now, 'expect' should be used for expectations:

 http://rspec.info/blog/2012/06/rspecs-new-expectation-syntax/
As far as I know, the only reason for that is that "should" is monkey patched on every class. This can cause problems if "method_missing" is implemented. D doesn't have this problem since "should" is not monkey patched on anything. In this code "should" is implemented like a free function and UFCS is used to get the same syntax as in Ruby. If there are problems with opDispatch/alias this (the D version of method_missing) then the regular function call syntax can be used. -- /Jacob Carlborg
Jul 03 2015