www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - CURL Wrapper: Congratulations Next up: std.serialize

reply "dsimcha" <dsimcha yahoo.com> writes:
By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) 
has been accepted into Phobos.  Thanks to Jonas for his hard work 
and his persistence through the multiple rounds of review that it 
took to get this module up to Phobos's high and increasing 
quality standard.

Keep the good work coming.  Next in line, if it's ready, is Jacob 
Carlborg's std.serialize.  Jacob, please post here when you've 
got something ready to go.
Dec 26 2011
next sibling parent "jdrewsen" <jdrewsen nospam.com> writes:
On Tuesday, 27 December 2011 at 02:01:51 UTC, dsimcha wrote:
 By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) 
 has been accepted into Phobos.  Thanks to Jonas for his hard 
 work and his persistence through the multiple rounds of review 
 that it took to get this module up to Phobos's high and 
 increasing quality standard.

 Keep the good work coming.  Next in line, if it's ready, is 
 Jacob Carlborg's std.serialize.  Jacob, please post here when 
 you've got something ready to go.

I'll go fix the last issues mentioned in the final review thread and the do a pull request. Thanks to all who provided valuable feedback.
Dec 27 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-27 03:01, dsimcha wrote:
 By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been
 accepted into Phobos. Thanks to Jonas for his hard work and his
 persistence through the multiple rounds of review that it took to get
 this module up to Phobos's high and increasing quality standard.

 Keep the good work coming. Next in line, if it's ready, is Jacob
 Carlborg's std.serialize. Jacob, please post here when you've got
 something ready to go.

Project page (two tutorials): http://dsource.org/projects/orange Repository: https://github.com/jacob-carlborg/orange Documentation: http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.Serializer.html (Don't forget the "Package" tab) Unit tests are available in the "tests" directory. These unit tests are not like regular unit tests that test individual functions. These unit tests are on a higher level. The most important part to review is the "serialization" package. The rest is mostly utility modules. Running the unit tests: ./unittest.sh Use "make" to compile the library or create an executable using rdmd. A few things to think about that need to be resolved: * This is quite a large library and I really don't want to put it all into one module. I'm hoping it will be OK with a package * I would really like to keep the unit tests in their own modules because they're quite large and the modules are already large without the unit tests in them * The unit tests use a kind of mini-unit test framework. Should that be kept or removed? Note: The documentation is generate using D1, I don't think that should make a difference though. -- /Jacob Carlborg
Dec 28 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-28 19:43, dsimcha wrote:
 On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg wrote:
 Running the unit tests:
 ./unittest.sh

 Use "make" to compile the library or create an executable using rdmd.

 A few things to think about that need to be resolved:

 * This is quite a large library and I really don't want to put it all
 into one module. I'm hoping it will be OK with a package

So the package would be std.serialize?

Yeah, or std.serialization.
 * I would really like to keep the unit tests in their own modules
 because they're quite large and the modules are already large without
 the unit tests in them

Sounds reasonable. It goes against the Phobos convention, but it sounds like you have a good reason to.
 * The unit tests use a kind of mini-unit test framework. Should that
 be kept or removed?

I haven't looked at it yet, but if it's generally useful, maybe it should be extracted and exposed as part of Phobos. I'd say keep it for now but keep it private, and later make a proposal for a full review to make it a public, official part of Phobos.

I think it is, don't know what others think. What it does is it catches AssertErrors so other unit tests can continue to run and then gives a nice report at the end. -- /Jacob Carlborg
Dec 28 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-29 06:11, Jonathan M Davis wrote:
 On Wednesday, December 28, 2011 23:07:51 Jacob Carlborg wrote:
 I think it is, don't know what others think. What it does is it catches
 AssertErrors so other unit tests can continue to run and then gives a
 nice report at the end.

I'm against it. I think that the compiler/runtime should be fixed so that each unit test block is run in a module even if one fails. That would solve the problem quite nicely IMHO, and that's already _supposed_ to be how it works. It just isn't properly implemented in that regard yet. And I'm against unittest blocks running any code after a single failure. So, I don't think that any additional unit testing framework is necessary. - Jonathan M Davis

Then the compiler need as well to collect all possible asserts and then somehow present them to the user. My library implementation already does this. Since this is completely implemented in library code it would be possible to have different formatters, a basic for outputting to the console and a more advanced with HTML output. Less important but can be quite useful sometimes is that with my framework you add contexts to the unit tests, just like RSpec for those familiar with it. If you haven't already, I suggest you take a look at the documentation for the unit test framework: http://dl.dropbox.com/u/18386187/orange_docs/orange.test.UnitTester.html For example: import orange.test.UnitTester; int sum (int x, int y) { return x * y; } unittest () { describe("sum") in { it("should return the sum of the two given arguments") in { assert(sum(1, 2) == 3); } } } void main () { run; } If a test fails the framework will print out the context, the stack trace and a snippet from the failing test, something like this: sum - should return the sum of the given arguments Failures: 1) sum should return the sum of the given arguments # main.d:44 Stack trace: tango.core.Exception.AssertException main(44): Assertion failure describe("sum") in { it("should return the sum of the given arguments") in { assert(sum(1, 2) == 3); }; }; 1 test, 1 failure -- /Jacob Carlborg
Dec 29 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-12-29 13:59, Jakob Ovrum wrote:
 On Thursday, 29 December 2011 at 12:49:55 UTC, Jacob Carlborg wrote:
 For example:

 import orange.test.UnitTester;

 int sum (int x, int y)
 {
 return x * y;
 }

 unittest ()
 {
 describe("sum") in {
 it("should return the sum of the two given arguments") in {
 assert(sum(1, 2) == 3);
 }
 }
 }

 void main ()
 {
 run;
 }

 If a test fails the framework will print out the context, the stack
 trace and a snippet from the failing test, something like this:

 sum
 - should return the sum of the given arguments

 Failures:
 1) sum should return the sum of the given arguments
 # main.d:44
 Stack trace:
 tango.core.Exception.AssertException main(44): Assertion failure


 describe("sum") in {
 it("should return the sum of the given arguments") in {
 assert(sum(1, 2) == 3);
 };
 };

 1 test, 1 failure

Operator overloading abuse, ahoy!

Yeah, I know. The "describe" and "it" functions can be changed to take the delegate as an argument instead. -- /Jacob Carlborg
Dec 29 2011
prev sibling parent reply Tobias Pankrath <tobias pankrath.net> writes:
When I first came to D, I read "unittest support" and thought that would 
be nice. But after I tried it, I realized it sucks and wrote something
similar to Jacobs unittest framework.

 I'm against it. I think that the compiler/runtime should be fixed so that
 each unit test block is run in a module even if one fails. That would
 solve the problem quite nicely IMHO, 

descriptions of the unit test that failed? That other unit tests in the module will not run after an error? That you can't run some specific unit-tests only?
 And
 I'm against unittest blocks running any code after a single failure. 

you have big modules. Imagine a high level unit-tests fails and you can't see the failure in the low level helper functions that nails down the error. However every library implemented unit test framework could just stop after the first failure, if configured properly. Should not be the default though.
 So, I
 don't think that any additional unit testing framework is necessary.

I really think it is and will use one for my D code. Since both worlds could live together peacefully there is absolutely no reason not to include one in phobos.
Dec 30 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-30 19:49, Jonathan M Davis wrote:
 On Friday, December 30, 2011 13:41:37 Tobias Pankrath wrote:
 I really think it is and will use one for my D code. Since both worlds
 could live together peacefully there is absolutely no reason not to include
 one in phobos.

It's one thing to use a fancier framework on your own. It's quite another to put it in the standard library. D's unit testing framework is designed to be straightforward and simple. On the whole, it does the job quite well. And once the issue of not running subsequent unittest blocks within a module after a failure in that module is fixed, I see no benefit from adding any additional library support. It just complicates things further. - Jonathan M Davis

Will that be able to give a proper report of all failed tests in a nice format? -- /Jacob Carlborg
Dec 30 2011
next sibling parent reply Tobias Pankrath <tobias pankrath.net> writes:
 
 I think that the AssertError's message (which includes the file and line
 number of the failure) and its stack trace are plenty. It's exactly what
 you need and nothing else.
 
 - Jonathan M Davis

I want to have such a summary. What's about running only certain unittests?
Dec 31 2011
next sibling parent Tobias Pankrath <tobias pankrath.net> writes:
Jonathan M Davis wrote:

 On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the file and
 line number of the failure) and its stack trace are plenty. It's
 exactly what you need and nothing else.
 
 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library.

I do see any reason not to put in the standard library. On the contrary: not having one in the standard library prohibits phobos modules to use an advanced solution.
 If you want that sort of summary, you probably want it printing stuff out
 on success too, and that definitely goes against how the built-in
 framework works (since it follows the typical unix approach of failure
 printing out stuff and success printing nothing). 

If I want to run the program only for the units tests, yes. But this does not touch the questions, whether or not to put a unit test framework in the stdlib.
 So, I think that that
 really makes more sense as a 3rd party solution rather than as part of the
 standard library. 

Since unit tests found their way in the language, I really see no reason, why they are no fit for the standard lib. What does a library qualifiy for phobos?
 And in general, 3rd party solutions are more likely to
 be customizable in a way you'd like rather than picking a single way of
 doing things.

Quality of other standard libraries shouldn't be an upper bound for phobos.
 D's unit test framework isn't designed that way at this point. 

way to run code at startup.
 You need
 named unit tests for that to really make sense. It could theoretically be
 added and would be nice, but that would require changes to the language
 (though fortunately, they would be backwards compatible changes). So, we
 may see that eventually but not right now. At this point, the closest that
 you get to that is to unit test each of your modules separately rather
 than all at once. 

solution for this.
 So, I'm not sure how common being
 able to really run a single unit test is anyway. 

I always run all of them and if one fails, I'll want to run those that failed to examine why they failed. I agree that we shouldn't add complexitiy to the buildin language unit tests, apart from some additions that you mention. But we should define a way, how an advanced unit test framework should operate with the core feature and of course they should work together. And since it is something many want to use (look at unit tests frameworks in other languages), it does have its place in the stdlib. PS: My newsreader (the KDE newsreader from kontact) seems to kill threading. Does anyone know how to change this without changing the newsreader?
Dec 31 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-31 11:37, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the file and line
 number of the failure) and its stack trace are plenty. It's exactly what
 you need and nothing else.

 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library. There's nothing wrong with 3rd party solutions which give additional functionality, but D's unit test framework is designed to be minimilistic, and I don't think that adding anything beyond what it does now in terms of summary makes any sense. The only major issue in that regard IMHO is the fact that no further unittest blocks within a module are run after one fails. Even if it did, I still don't think that a fancier summary would be worth having - especially in the standard library. If you want that sort of summary, you probably want it printing stuff out on success too, and that definitely goes against how the built-in framework works (since it follows the typical unix approach of failure printing out stuff and success printing nothing). So, I think that that really makes more sense as a 3rd party solution rather than as part of the standard library. And in general, 3rd party solutions are more likely to be customizable in a way you'd like rather than picking a single way of doing things.
 What's about running only certain unittests?

D's unit test framework isn't designed that way at this point. You need named unit tests for that to really make sense. It could theoretically be added and would be nice, but that would require changes to the language (though fortunately, they would be backwards compatible changes). So, we may see that eventually but not right now. At this point, the closest that you get to that is to unit test each of your modules separately rather than all at once. And actually, even major unit testing frameworks such as JUnit often end up running all of the unit tests within a module/file when you tell them to run a single unit test (probably in part since one unit test can theoretically affect the ones that follow it, and probably in part due to how they're implemented). So, I'm not sure how common being able to really run a single unit test is anyway. It would be a nice addition, and we may get it eventually, but it's not going to happen right now. - Jonathan M Davis

It would be possible to implement named unit tests only in library code. It would not have as nice syntax as if it was implemented in the language but still possible. In Ruby on Rails I run single unit tests all the time. Why would I run all the unit tests, which can take five minutes, when I just can run one unit test and it takes just one second? When your doing test/behavior driven development (T/BDD) it's certainly nice to be able to run single unit tests, because you run it all the time. -- /Jacob Carlborg
Dec 31 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-31 22:01, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 16:04:12 Jacob Carlborg wrote:
 It would be possible to implement named unit tests only in library code.
 It would not have as nice syntax as if it was implemented in the
 language but still possible.

 In Ruby on Rails I run single unit tests all the time. Why would I run
 all the unit tests, which can take five minutes, when I just can run one
 unit test and it takes just one second?

 When your doing test/behavior driven development (T/BDD) it's certainly
 nice to be able to run single unit tests, because you run it all the time.

Yes. I agree that it would be nice, but for it to be done at all cleanly, the language, compiler, and druntime need to be improved to make it possible. However, at least syntactically, such changes should be completely backwards compatible, so they can be added at a future date. Regardless, I don't think that it's a problem that Phobos should be trying to solve. - Jonathan M Davis

Ok, if you would rather have all this in the language I would say no do that. But I know other people in the community that usually prefer to do a library solution if possible. -- /Jacob Carlborg
Jan 01 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-01-02 00:14, Jonathan M Davis wrote:
 On Sunday, January 01, 2012 15:35:00 Jacob Carlborg wrote:
 Ok, if you would rather have all this in the language I would say no do
 that. But I know other people in the community that usually prefer to do
 a library solution if possible.

If all we're talking about is named unit tests and running all of the unittest blocks within a module even if one fails, those aren't huge changes to the language. Both have been discussed before, and the only reason that the latter hasn't been implemented yet is that it requires changes to the compiler. So, I don't see any reason to make those a library solution. If we're talking something massively more complicated than that, then yes, a library solution starts making more sense. I also think that it then doesn't necessarily make sense to put it in the standard library. If the issue is how the tests print out on failure, that could probably be done in the language as well, depending on what you were talking about changing. If you want to do something massively complicated, however, then you'd probably have to come up with a library solution, but again, I see no reason to have it in the standard library if you're doing anything fancy with how the test failures print out. - Jonathan M Davis

I see it as three things I want done. * Named unit tests * Continue to run unit tests after a failed one * A nice report at the end, preferably configurable It would be nice if the first two were implemented in the language. If the third one is implemented in the runtime it could perhaps be configurable. One implements an interface, or similar, and can output the unit test report how he/she wants. -- /Jacob Carlborg
Jan 01 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-31 11:37, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the file and line
 number of the failure) and its stack trace are plenty. It's exactly what
 you need and nothing else.

 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library. There's nothing wrong with 3rd party solutions which give additional functionality, but D's unit test framework is designed to be minimilistic, and I don't think that adding anything beyond what it does now in terms of summary makes any sense. The only major issue in that regard IMHO is the fact that no further unittest blocks within a module are run after one fails. Even if it did, I still don't think that a fancier summary would be worth having - especially in the standard library.

BTW, what would be so wrong if the unit tests for the standard library displayed a nice report when finished? -- /Jacob Carlborg
Dec 31 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-31 21:56, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
 BTW, what would be so wrong if the unit tests for the standard library
 displayed a nice report when finished?

My primary issue here is that I don't think that we should be adding stuff to Phobos which is essentially a new unit test framework on top of the built in one. If 3rd party stuff wants to do that. Fine. But the standard library should use the standard facilities. If the standard facilities aren't sufficient, then they should be improved.

First, that's how programming works. You build new abstractions on top of existing ones. Second, I would called the built in support for unit testing in D a "unit test framework". If add a unit test framework to Phobos it would be standard facilities and it wouldn't be a problem.
 As for a "nice report," I don't see anything wrong with just using the stack
 traces (which include the file, line number, and error message of the assertion
 failure). That's all the information that's needed. Anything else is
 superfluous IMHO. Now, if there were something nicer that could be generally
 agreed upon and added to druntime such that the standard unit test facilities
 used it, then fine. I don't see any point to it, but at least in that case, the
 standard library is still using the standard unit test framework. What I
 really don't want to see is Phobos essentially building a new unit test
 framework on top of the existing one. Any issues that need to be addressed
 with the unit test framework for the standard library should be addressed in
 the standard framework. Any additional framework stuff should be left to 3rd
 parties.

 - Jonathan M Davis

What's wrong with being able to run the unit tests from your editor, have the unit test framework output HTML (or similar), displayed in your editor and then you can click on links in the stack trace to get to the source code. If you don't see why that's useful that we can just end this discussion now. -- /Jacob Carlborg
Jan 01 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-01-02 00:28, Jonathan M Davis wrote:
 On Sunday, January 01, 2012 15:31:18 Jacob Carlborg wrote:
 What's wrong with being able to run the unit tests from your editor,
 have the unit test framework output HTML (or similar), displayed in your
 editor and then you can click on links in the stack trace to get to the
 source code. If you don't see why that's useful that we can just end
 this discussion now.

If you want fancier unit test facilities in your own code. Fine. I just don't think that they should be in the standard library. I think that on the whole, D's built-in unit test framework works fine as it is (barring a few tweaks such as making it so that all unittest blocks within a module run), and that further additions are a needless complication to the standard unit testing facilities and better left to 3rd party solutions where you can do whatever you want with them. - Jonathan M Davis

I'm ending the discussion here. -- /Jacob Carlborg
Jan 01 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-01-01 01:57, Andrew Wiley wrote:
 On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis<jmdavisProg gmx.com>  wrote:
 On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
 On 2011-12-31 11:37, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the file and
 line number of the failure) and its stack trace are plenty. It's
 exactly what you need and nothing else.

 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library. There's nothing wrong with 3rd party solutions which give additional functionality, but D's unit test framework is designed to be minimilistic, and I don't think that adding anything beyond what it does now in terms of summary makes any sense. The only major issue in that regard IMHO is the fact that no further unittest blocks within a module are run after one fails. Even if it did, I still don't think that a fancier summary would be worth having - especially in the standard library.

BTW, what would be so wrong if the unit tests for the standard library displayed a nice report when finished?

My primary issue here is that I don't think that we should be adding stuff to Phobos which is essentially a new unit test framework on top of the built in one. If 3rd party stuff wants to do that. Fine. But the standard library should use the standard facilities. If the standard facilities aren't sufficient, then they should be improved.

The counterargument is that the language doesn't really provide a framework - it actually provides anonymous parameterless global functions that will be run before main is invoked if code is compiled with -unittest. That isn't considered a framework in any language I've ever used, but it adds just enough functionality to allow a well-integrated fully-featured library solution. Would making such a library solution part of the standard library really be a problem? I'm mostly ambivalent on this issue because I haven't had time to look closely at the proposed framework, but your argument seems to be that all unittesting functionality needs to be built into the language. I don't think that should be necessary or required.
 As for a "nice report," I don't see anything wrong with just using the stack
 traces (which include the file, line number, and error message of the assertion
 failure). That's all the information that's needed. Anything else is
 superfluous IMHO. Now, if there were something nicer that could be generally
 agreed upon and added to druntime such that the standard unit test facilities
 used it, then fine. I don't see any point to it, but at least in that case, the
 standard library is still using the standard unit test framework. What I
 really don't want to see is Phobos essentially building a new unit test
 framework on top of the existing one. Any issues that need to be addressed
 with the unit test framework for the standard library should be addressed in
 the standard framework. Any additional framework stuff should be left to 3rd
 parties.

As I said above, I wouldn't consider what we have to be a framework, but it's definitely enough to build an excellent library solution on top of. As for a report, the problem is that an assertion error isn't what you want when you're running, say, a continuous integration server (or, say, a pull request tester). What you really want is a detailed explanation of what unittests broke, what the tests were testing, and how the result differed from what was expected. You want to be able to have a reasonable idea of what went wrong *without* having to look at someone else's code and figure out exactly what they're testing every time.

Exactly, I agree. -- /Jacob Carlborg
Jan 01 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-12-31 04:35, Jonathan M Davis wrote:
 On Friday, December 30, 2011 21:38:07 Jacob Carlborg wrote:
 On 2011-12-30 19:49, Jonathan M Davis wrote:
 On Friday, December 30, 2011 13:41:37 Tobias Pankrath wrote:
 I really think it is and will use one for my D code. Since both worlds
 could live together peacefully there is absolutely no reason not to
 include one in phobos.

It's one thing to use a fancier framework on your own. It's quite another to put it in the standard library. D's unit testing framework is designed to be straightforward and simple. On the whole, it does the job quite well. And once the issue of not running subsequent unittest blocks within a module after a failure in that module is fixed, I see no benefit from adding any additional library support. It just complicates things further. - Jonathan M Davis

Will that be able to give a proper report of all failed tests in a nice format?

I think that the AssertError's message (which includes the file and line number of the failure) and its stack trace are plenty. It's exactly what you need and nothing else. - Jonathan M Davis

Yes but what happens when there are many failed tests, i.e. may AssertErrors that have been thrown? It will just print all after each other and you have to count them yourself if you want to know how many failed tests there are? -- /Jacob Carlborg
Dec 31 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-12-31 21:57, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 15:48:16 Jacob Carlborg wrote:
 Yes but what happens when there are many failed tests, i.e. may
 AssertErrors that have been thrown? It will just print all after each
 other and you have to count them yourself if you want to know how many
 failed tests there are?

What does the number of failures really matter? You just need to know which ones failed and where. The AssertErrors give you that. - Jonathan M Davis

It can be nice for statistics. And again, how will it be displayed, just all AssertErrors after each other? -- /Jacob Carlborg
Jan 01 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-12-28 22:19, jdrewsen wrote:
 On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg wrote:
 On 2011-12-27 03:01, dsimcha wrote:
 By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been
 accepted into Phobos. Thanks to Jonas for his hard work and his
 persistence through the multiple rounds of review that it took to get
 this module up to Phobos's high and increasing quality standard.

 Keep the good work coming. Next in line, if it's ready, is Jacob
 Carlborg's std.serialize. Jacob, please post here when you've got
 something ready to go.

Project page (two tutorials): http://dsource.org/projects/orange Repository: https://github.com/jacob-carlborg/orange Documentation: http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.Serializer.html (Don't forget the "Package" tab) Unit tests are available in the "tests" directory. These unit tests are not like regular unit tests that test individual functions. These unit tests are on a higher level. The most important part to review is the "serialization" package. The rest is mostly utility modules.

After I quick look: I think that all other modules/packages under the orange package should either be integrated into other existing phobos modules or as new phobos modules since they are really not serialization specific.

I agree with that.
 Furthermore I think that all suppport for tango in the code should be
 stripped before going into phobos.

 /Jonas

That is obvious. I don't know if this have been clear or not but I've requested a pre-review. A review that should be as a regular review but before I do a Phobos integration. If it gets accepted into Phobos I'll do the necessary modifications (remove all Tango related code and so on) to integrate it into Phobos and then we can have a second review. I don't want to waste time on removing the Tango support if I don't know for sure that it will be accepted. -- /Jacob Carlborg
Dec 28 2011
prev sibling next sibling parent "dsimcha" <dsimcha yahoo.com> writes:
On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg 
wrote:
 Running the unit tests:
 ./unittest.sh

 Use "make" to compile the library or create an executable using 
 rdmd.

 A few things to think about that need to be resolved:

 * This is quite a large library and I really don't want to put 
 it all into one module. I'm hoping it will be OK with a package

So the package would be std.serialize?
 * I would really like to keep the unit tests in their own 
 modules because they're quite large and the modules are already 
 large without the unit tests in them

Sounds reasonable. It goes against the Phobos convention, but it sounds like you have a good reason to.
 * The unit tests use a kind of mini-unit test framework. Should 
 that be kept or removed?

I haven't looked at it yet, but if it's generally useful, maybe it should be extracted and exposed as part of Phobos. I'd say keep it for now but keep it private, and later make a proposal for a full review to make it a public, official part of Phobos.
 Note:

 The documentation is generate using D1, I don't think that 
 should make a difference though.

Dec 28 2011
prev sibling next sibling parent "jdrewsen" <jdrewsen nospam.com> writes:
On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg 
wrote:
 On 2011-12-27 03:01, dsimcha wrote:
 By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) 
 has been
 accepted into Phobos. Thanks to Jonas for his hard work and his
 persistence through the multiple rounds of review that it took 
 to get
 this module up to Phobos's high and increasing quality 
 standard.

 Keep the good work coming. Next in line, if it's ready, is 
 Jacob
 Carlborg's std.serialize. Jacob, please post here when you've 
 got
 something ready to go.

Project page (two tutorials): http://dsource.org/projects/orange Repository: https://github.com/jacob-carlborg/orange Documentation: http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.Serializer.html (Don't forget the "Package" tab) Unit tests are available in the "tests" directory. These unit tests are not like regular unit tests that test individual functions. These unit tests are on a higher level. The most important part to review is the "serialization" package. The rest is mostly utility modules.

After I quick look: I think that all other modules/packages under the orange package should either be integrated into other existing phobos modules or as new phobos modules since they are really not serialization specific. Furthermore I think that all suppport for tango in the code should be stripped before going into phobos. /Jonas
Dec 28 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, December 28, 2011 23:07:51 Jacob Carlborg wrote:
 I think it is, don't know what others think. What it does is it catches
 AssertErrors so other unit tests can continue to run and then gives a
 nice report at the end.

I'm against it. I think that the compiler/runtime should be fixed so that each unit test block is run in a module even if one fails. That would solve the problem quite nicely IMHO, and that's already _supposed_ to be how it works. It just isn't properly implemented in that regard yet. And I'm against unittest blocks running any code after a single failure. So, I don't think that any additional unit testing framework is necessary. - Jonathan M Davis
Dec 28 2011
prev sibling next sibling parent Brad Anderson <eco gnuk.net> writes:
--f46d0435c12a18c07b04b535c3c0
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Dec 28, 2011 at 10:11 PM, Jonathan M Davis <jmdavisProg gmx.com>wrote:

 On Wednesday, December 28, 2011 23:07:51 Jacob Carlborg wrote:
 I think it is, don't know what others think. What it does is it catches
 AssertErrors so other unit tests can continue to run and then gives a
 nice report at the end.

I'm against it. I think that the compiler/runtime should be fixed so that each unit test block is run in a module even if one fails. That would solve the problem quite nicely IMHO, and that's already _supposed_ to be how it works. It just isn't properly implemented in that regard yet. And I'm against unittest blocks running any code after a single failure. So, I don't think that any additional unit testing framework is necessary. - Jonathan M Davis

Forgive me if this is a silly question but a conversation in IRC got me wondering if compiler could check for shared/__gshared use (and any other thread unsafe operation) in each unittest and run those that aren't using them concurrently? Obviously not all at once but at least more than one at a time (perhaps set through user configuration). Regards, Brad --f46d0435c12a18c07b04b535c3c0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Wed, Dec 28, 2011 at 10:11 PM, Jonathan M Davis <span dir=3D"ltr">&lt;<a= href=3D"mailto:jmdavisProg gmx.com">jmdavisProg gmx.com</a>&gt;</span> wro= te:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D"im">On Wednesday, December 28, 2011 23:07:51 Jacob Carlborg w= rote:<br> &gt; I think it is, don&#39;t know what others think. What it does is it ca= tches<br> &gt; AssertErrors so other unit tests can continue to run and then gives a<= br> &gt; nice report at the end.<br> <br> </div>I&#39;m against it. I think that the compiler/runtime should be fixed= so that each<br> unit test block is run in a module even if one fails. That would solve the<= br> problem quite nicely IMHO, and that&#39;s already _supposed_ to be how it w= orks.<br> It just isn&#39;t properly implemented in that regard yet. And I&#39;m agai= nst<br> unittest blocks running any code after a single failure. So, I don&#39;t th= ink<br> that any additional unit testing framework is necessary.<br> <span class=3D"HOEnZb"><font color=3D"#888888"><br> - Jonathan M Davis<br> </font></span></blockquote></div><br><div>Forgive me if this is a silly que= stion but a conversation in IRC got me wondering if compiler could check fo= r shared/__gshared use=A0(and any other thread unsafe operation)=A0in each = unittest and run those that aren&#39;t using them=A0concurrently? Obviously= not all at once but at least more than one at a time (perhaps set through = user configuration).</div> <div><br></div><div>Regards,</div><div>Brad</div> --f46d0435c12a18c07b04b535c3c0--
Dec 28 2011
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Thursday, 29 December 2011 at 07:07:10 UTC, Brad Anderson 
wrote:
 Forgive me if this is a silly question but a conversation in 
 IRC got me
 wondering if compiler could check for shared/__gshared use (and 
 any other
 thread unsafe operation) in each unittest and run those that 
 aren't using
 them concurrently? Obviously not all at once but at least more 
 than one at
 a time (perhaps set through user configuration).

 Regards,
 Brad

The unittest code can ultimately call into both C and D code which source is not known. Such code would need to cause the unittest to be excluded from concurrent execution, in light of this I think it's questionable if such an analysis would be worth implementing. Perhaps look at some example unittest suites that take a long time and look at whether these examples can be proven to be thread-safe.
Dec 28 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, December 29, 2011 00:06:57 Brad Anderson wrote:
 Forgive me if this is a silly question but a conversation in IRC got me
 wondering if compiler could check for shared/__gshared use (and any other
 thread unsafe operation) in each unittest and run those that aren't using
 them concurrently? Obviously not all at once but at least more than one at
 a time (perhaps set through user configuration).

That could result in non-deterministic failures. Granted, unittest blocks should usually be independent, but there's nothing that guarantees that they are, so running them in a non-derministic order could result in non- deterministic failures. Also, what about unit tests for stuff like std.pararellism or std.concurrency - anything that actually specifically uses threads in its implementation? Running tests in their own threads could cause issues with that. Each module in Phobos has its own executable for running its unit tests, and there has been talk of making _those_ parallelizable, and I think that parallelizing things on that level makes a lot more sense than trying to build it into the language. - Jonathan M Davis
Dec 28 2011
prev sibling next sibling parent "jdrewsen" <jdrewsen nospam.com> writes:
On Wednesday, 28 December 2011 at 22:21:09 UTC, Jacob Carlborg 
wrote:
 On 2011-12-28 22:19, jdrewsen wrote:
 On Wednesday, 28 December 2011 at 16:01:50 UTC, Jacob Carlborg 
 wrote:
 On 2011-12-27 03:01, dsimcha wrote:
 By a vote of 14-0, Jonas Drewsen's CURL wrapper 
 (std.net.curl) has been
 accepted into Phobos. Thanks to Jonas for his hard work and 
 his
 persistence through the multiple rounds of review that it 
 took to get
 this module up to Phobos's high and increasing quality 
 standard.

 Keep the good work coming. Next in line, if it's ready, is 
 Jacob
 Carlborg's std.serialize. Jacob, please post here when 
 you've got
 something ready to go.

Project page (two tutorials): http://dsource.org/projects/orange Repository: https://github.com/jacob-carlborg/orange Documentation: http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.Serializer.html (Don't forget the "Package" tab) Unit tests are available in the "tests" directory. These unit tests are not like regular unit tests that test individual functions. These unit tests are on a higher level. The most important part to review is the "serialization" package. The rest is mostly utility modules.

After I quick look: I think that all other modules/packages under the orange package should either be integrated into other existing phobos modules or as new phobos modules since they are really not serialization specific.

I agree with that.
 Furthermore I think that all suppport for tango in the code 
 should be
 stripped before going into phobos.

 /Jonas

That is obvious. I don't know if this have been clear or not but I've requested a pre-review. A review that should be as a regular review but before I do a Phobos integration. If it gets accepted into Phobos I'll do the necessary modifications (remove all Tango related code and so on) to integrate it into Phobos and then we can have a second review. I don't want to waste time on removing the Tango support if I don't know for sure that it will be accepted.

Ok that wasn't clear to me. I've updated the review queue: http://prowiki.org/wiki4d/wiki.cgi?ReviewQueue /Jonas
Dec 29 2011
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-12-27 03:01, dsimcha wrote:
 By a vote of 14-0, Jonas Drewsen's CURL wrapper (std.net.curl) has been
 accepted into Phobos. Thanks to Jonas for his hard work and his
 persistence through the multiple rounds of review that it took to get
 this module up to Phobos's high and increasing quality standard.

 Keep the good work coming. Next in line, if it's ready, is Jacob
 Carlborg's std.serialize. Jacob, please post here when you've got
 something ready to go.

If we're going with my serialization library as the next item to review, should we have a review manager and a new thread? -- /Jacob Carlborg
Dec 29 2011
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Thursday, 29 December 2011 at 12:49:55 UTC, Jacob Carlborg 
wrote:
 For example:

 import orange.test.UnitTester;

 int sum (int x, int y)
 {
    return x * y;
 }

 unittest ()
 {
    describe("sum") in {
         it("should return the sum of the two given arguments") 
 in {
              assert(sum(1, 2) == 3);
         }
    }
 }

 void main ()
 {
    run;
 }

 If a test fails the framework will print out the context, the 
 stack trace and a snippet from the failing test, something like 
 this:

 sum
   - should return the sum of the given arguments

 Failures:
     1) sum should return the sum of the given arguments
        # main.d:44
        Stack trace:
        tango.core.Exception.AssertException main(44): Assertion 
 failure


 describe("sum") in {
 	it("should return the sum of the given arguments") in {
 		assert(sum(1, 2) == 3);
 	};
 };

 1 test, 1 failure

Operator overloading abuse, ahoy!
Dec 29 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, December 30, 2011 13:41:37 Tobias Pankrath wrote:
 When I first came to D, I read "unittest support" and thought that would
 be nice. But after I tried it, I realized it sucks and wrote something
 similar to Jacobs unittest framework.
 
 I'm against it. I think that the compiler/runtime should be fixed so
 that
 each unit test block is run in a module even if one fails. That would
 solve the problem quite nicely IMHO,

Which problem? That you can't get a good summary or backtrace? No descriptions of the unit test that failed? That other unit tests in the module will not run after an error? That you can't run some specific unit-tests only?

You get a proper error message backtrace on unit test failure. The only problem is that once a single unittest block within a module fails, no further unittest blocks within that module are run (unittest blocks from subsequent modules are run but no more from the one with the failure). As such, once you get a single failure within a module, you can't see any further failures. That _is_ a problem, but it's the only one that I'm aware of. And that needs to be fixed in the runtime, _not_ by adding more complex library support. It's a known issue which requires some additional support by the compiler (additional hooks of some kind IIRC) for the runtime to be fixed, but it is the runtime that should be fixed rather than trying to fix it with a library.
 And
 I'm against unittest blocks running any code after a single failure.

On the other hand, I think this is absolutely necessary, especially if you have big modules. Imagine a high level unit-tests fails and you can't see the failure in the low level helper functions that nails down the error. However every library implemented unit test framework could just stop after the first failure, if configured properly. Should not be the default though.
 So, I
 don't think that any additional unit testing framework is necessary.

I really think it is and will use one for my D code. Since both worlds could live together peacefully there is absolutely no reason not to include one in phobos.

It's one thing to use a fancier framework on your own. It's quite another to put it in the standard library. D's unit testing framework is designed to be straightforward and simple. On the whole, it does the job quite well. And once the issue of not running subsequent unittest blocks within a module after a failure in that module is fixed, I see no benefit from adding any additional library support. It just complicates things further. - Jonathan M Davis
Dec 30 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, December 30, 2011 21:38:07 Jacob Carlborg wrote:
 On 2011-12-30 19:49, Jonathan M Davis wrote:
 On Friday, December 30, 2011 13:41:37 Tobias Pankrath wrote:
 I really think it is and will use one for my D code. Since both worlds
 could live together peacefully there is absolutely no reason not to
 include one in phobos.

It's one thing to use a fancier framework on your own. It's quite another to put it in the standard library. D's unit testing framework is designed to be straightforward and simple. On the whole, it does the job quite well. And once the issue of not running subsequent unittest blocks within a module after a failure in that module is fixed, I see no benefit from adding any additional library support. It just complicates things further. - Jonathan M Davis

Will that be able to give a proper report of all failed tests in a nice format?

I think that the AssertError's message (which includes the file and line number of the failure) and its stack trace are plenty. It's exactly what you need and nothing else. - Jonathan M Davis
Dec 30 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the file and line
 number of the failure) and its stack trace are plenty. It's exactly what
 you need and nothing else.
 
 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library. There's nothing wrong with 3rd party solutions which give additional functionality, but D's unit test framework is designed to be minimilistic, and I don't think that adding anything beyond what it does now in terms of summary makes any sense. The only major issue in that regard IMHO is the fact that no further unittest blocks within a module are run after one fails. Even if it did, I still don't think that a fancier summary would be worth having - especially in the standard library. If you want that sort of summary, you probably want it printing stuff out on success too, and that definitely goes against how the built-in framework works (since it follows the typical unix approach of failure printing out stuff and success printing nothing). So, I think that that really makes more sense as a 3rd party solution rather than as part of the standard library. And in general, 3rd party solutions are more likely to be customizable in a way you'd like rather than picking a single way of doing things.
 What's about running only certain unittests?

D's unit test framework isn't designed that way at this point. You need named unit tests for that to really make sense. It could theoretically be added and would be nice, but that would require changes to the language (though fortunately, they would be backwards compatible changes). So, we may see that eventually but not right now. At this point, the closest that you get to that is to unit test each of your modules separately rather than all at once. And actually, even major unit testing frameworks such as JUnit often end up running all of the unit tests within a module/file when you tell them to run a single unit test (probably in part since one unit test can theoretically affect the ones that follow it, and probably in part due to how they're implemented). So, I'm not sure how common being able to really run a single unit test is anyway. It would be a nice addition, and we may get it eventually, but it's not going to happen right now. - Jonathan M Davis
Dec 31 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
 On 2011-12-31 11:37, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the file and
 line number of the failure) and its stack trace are plenty. It's
 exactly what you need and nothing else.
 
 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library. There's nothing wrong with 3rd party solutions which give additional functionality, but D's unit test framework is designed to be minimilistic, and I don't think that adding anything beyond what it does now in terms of summary makes any sense. The only major issue in that regard IMHO is the fact that no further unittest blocks within a module are run after one fails. Even if it did, I still don't think that a fancier summary would be worth having - especially in the standard library.

BTW, what would be so wrong if the unit tests for the standard library displayed a nice report when finished?

My primary issue here is that I don't think that we should be adding stuff to Phobos which is essentially a new unit test framework on top of the built in one. If 3rd party stuff wants to do that. Fine. But the standard library should use the standard facilities. If the standard facilities aren't sufficient, then they should be improved. As for a "nice report," I don't see anything wrong with just using the stack traces (which include the file, line number, and error message of the assertion failure). That's all the information that's needed. Anything else is superfluous IMHO. Now, if there were something nicer that could be generally agreed upon and added to druntime such that the standard unit test facilities used it, then fine. I don't see any point to it, but at least in that case, the standard library is still using the standard unit test framework. What I really don't want to see is Phobos essentially building a new unit test framework on top of the existing one. Any issues that need to be addressed with the unit test framework for the standard library should be addressed in the standard framework. Any additional framework stuff should be left to 3rd parties. - Jonathan M Davis
Dec 31 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, December 31, 2011 15:48:16 Jacob Carlborg wrote:
 Yes but what happens when there are many failed tests, i.e. may
 AssertErrors that have been thrown? It will just print all after each
 other and you have to count them yourself if you want to know how many
 failed tests there are?

What does the number of failures really matter? You just need to know which ones failed and where. The AssertErrors give you that. - Jonathan M Davis
Dec 31 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, December 31, 2011 16:04:12 Jacob Carlborg wrote:
 It would be possible to implement named unit tests only in library code.
 It would not have as nice syntax as if it was implemented in the
 language but still possible.
 
 In Ruby on Rails I run single unit tests all the time. Why would I run
 all the unit tests, which can take five minutes, when I just can run one
 unit test and it takes just one second?
 
 When your doing test/behavior driven development (T/BDD) it's certainly
 nice to be able to run single unit tests, because you run it all the time.

Yes. I agree that it would be nice, but for it to be done at all cleanly, the language, compiler, and druntime need to be improved to make it possible. However, at least syntactically, such changes should be completely backwards compatible, so they can be added at a future date. Regardless, I don't think that it's a problem that Phobos should be trying to solve. - Jonathan M Davis
Dec 31 2011
prev sibling next sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis <jmdavisProg gmx.com> wrote:
 On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
 On 2011-12-31 11:37, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the file and
 line number of the failure) and its stack trace are plenty. It's
 exactly what you need and nothing else.

 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library. There's nothing wrong with 3rd party solutions which give additional functionality, but D's unit test framework is designed to be minimilistic, and I don't think that adding anything beyond what it does now in terms of summary makes any sense. The only major issue in that regard IMHO is the fact that no further unittest blocks within a module are run after one fails. Even if it did, I still don't think that a fancier summary would be worth having - especially in the standard library.

BTW, what would be so wrong if the unit tests for the standard library displayed a nice report when finished?

My primary issue here is that I don't think that we should be adding stuff to Phobos which is essentially a new unit test framework on top of the built in one. If 3rd party stuff wants to do that. Fine. But the standard library should use the standard facilities. If the standard facilities aren't sufficient, then they should be improved.

The counterargument is that the language doesn't really provide a framework - it actually provides anonymous parameterless global functions that will be run before main is invoked if code is compiled with -unittest. That isn't considered a framework in any language I've ever used, but it adds just enough functionality to allow a well-integrated fully-featured library solution. Would making such a library solution part of the standard library really be a problem? I'm mostly ambivalent on this issue because I haven't had time to look closely at the proposed framework, but your argument seems to be that all unittesting functionality needs to be built into the language. I don't think that should be necessary or required.
 As for a "nice report," I don't see anything wrong with just using the stack
 traces (which include the file, line number, and error message of the assertion
 failure). That's all the information that's needed. Anything else is
 superfluous IMHO. Now, if there were something nicer that could be generally
 agreed upon and added to druntime such that the standard unit test facilities
 used it, then fine. I don't see any point to it, but at least in that case, the
 standard library is still using the standard unit test framework. What I
 really don't want to see is Phobos essentially building a new unit test
 framework on top of the existing one. Any issues that need to be addressed
 with the unit test framework for the standard library should be addressed in
 the standard framework. Any additional framework stuff should be left to 3rd
 parties.

As I said above, I wouldn't consider what we have to be a framework, but it's definitely enough to build an excellent library solution on top of. As for a report, the problem is that an assertion error isn't what you want when you're running, say, a continuous integration server (or, say, a pull request tester). What you really want is a detailed explanation of what unittests broke, what the tests were testing, and how the result differed from what was expected. You want to be able to have a reasonable idea of what went wrong *without* having to look at someone else's code and figure out exactly what they're testing every time.
Dec 31 2011
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, December 31, 2011 18:57:14 Andrew Wiley wrote:
 On Sat, Dec 31, 2011 at 2:56 PM, Jonathan M Davis <jmdavisProg gmx.com> 

 On Saturday, December 31, 2011 16:06:49 Jacob Carlborg wrote:
 On 2011-12-31 11:37, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 11:05:58 Tobias Pankrath wrote:
 I think that the AssertError's message (which includes the
 file and
 line number of the failure) and its stack trace are plenty.
 It's
 exactly what you need and nothing else.
 
 - Jonathan M Davis

I want to have such a summary.

I don't see any reason to put that in the standard library. There's nothing wrong with 3rd party solutions which give additional functionality, but D's unit test framework is designed to be minimilistic, and I don't think that adding anything beyond what it does now in terms of summary makes any sense. The only major issue in that regard IMHO is the fact that no further unittest blocks within a module are run after one fails. Even if it did, I still don't think that a fancier summary would be worth having - especially in the standard library.

BTW, what would be so wrong if the unit tests for the standard library displayed a nice report when finished?

My primary issue here is that I don't think that we should be adding stuff to Phobos which is essentially a new unit test framework on top of the built in one. If 3rd party stuff wants to do that. Fine. But the standard library should use the standard facilities. If the standard facilities aren't sufficient, then they should be improved.

The counterargument is that the language doesn't really provide a framework - it actually provides anonymous parameterless global functions that will be run before main is invoked if code is compiled with -unittest. That isn't considered a framework in any language I've ever used, but it adds just enough functionality to allow a well-integrated fully-featured library solution. Would making such a library solution part of the standard library really be a problem? I'm mostly ambivalent on this issue because I haven't had time to look closely at the proposed framework, but your argument seems to be that all unittesting functionality needs to be built into the language. I don't think that should be necessary or required.

For the most part, I don't think that we need anything more. It should be fixed so that all unittest blocks within a module are run even if one fails. This should be fixed in the compiler and runtime rather than having to hack it together with a library solution, since it's supposed to work that way in the first place but doesn't because of some changes that need to be made to the compiler (some hooks of some kind are needed IIRC). Named unit tests would also be nice, but it would be _far_ cleaner to add those to the language than try and add them with a library solution (e.g. unittest(test_name){}). It would also be nice if it were possible to hook in something that would run specific unit tests, but again, such hooks would then need to be provided by the compiler and/or runtime - unless you want to simply create a function for each unit test and have each unittest block call a specific function? That's an ugly solution IMHO. All of those changes would be fairly minor in terms of how they affect how unit tests function and would be far cleaner IMHO to have in the language. I think that on some level, the unit test framework in the language has failed if we have to add library solutions on top of it to get such basic functionality.
 As for a "nice report," I don't see anything wrong with just using the
 stack traces (which include the file, line number, and error message of
 the assertion failure). That's all the information that's needed.
 Anything else is superfluous IMHO. Now, if there were something nicer
 that could be generally agreed upon and added to druntime such that the
 standard unit test facilities used it, then fine. I don't see any point
 to it, but at least in that case, the standard library is still using
 the standard unit test framework. What I really don't want to see is
 Phobos essentially building a new unit test framework on top of the
 existing one. Any issues that need to be addressed with the unit test
 framework for the standard library should be addressed in the standard
 framework. Any additional framework stuff should be left to 3rd
 parties.

As I said above, I wouldn't consider what we have to be a framework, but it's definitely enough to build an excellent library solution on top of. As for a report, the problem is that an assertion error isn't what you want when you're running, say, a continuous integration server (or, say, a pull request tester). What you really want is a detailed explanation of what unittests broke, what the tests were testing, and how the result differed from what was expected. You want to be able to have a reasonable idea of what went wrong *without* having to look at someone else's code and figure out exactly what they're testing every time.

I don't really get what you'd need beyond knowing which test failed. The AssertErrors tell you that. It would definitely be nicer if the unittest blocks had proper names to go with them, but they're not necessary for knowing which test failed. The file and line number give you that. I don't see how you're going to know what actually failed unless you actually look at the test. If the tests have names and are well named, then that gives you a better idea of what isn't working before you look at the code, but ultimately, you have to look at the code to know what went wrong. And if you can name unittest blocks, then that problem is solved anyway, since then you'd have a name for the unit test failure to give in any report that you wanted to create. - Jonathan M Davis
Dec 31 2011
parent Tobias Pankrath <tobias pankrath.net> writes:
Jonathan M Davis wrote:

 I think
 that on some level, the unit test framework in the language has failed if
 we have to add library solutions on top of it to get such basic
 functionality.

That's what we are saying: The unit test framework fails for us and a library solution is perfectly possible, even one that lets everyone use the build in functionality if he wants. You don't find it useful, but others do. I'd say: many others do. If something is considered useful by many it should have a place in the standard library.
Jan 01 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, January 01, 2012 15:35:00 Jacob Carlborg wrote:
 Ok, if you would rather have all this in the language I would say no do
 that. But I know other people in the community that usually prefer to do
 a library solution if possible.

If all we're talking about is named unit tests and running all of the unittest blocks within a module even if one fails, those aren't huge changes to the language. Both have been discussed before, and the only reason that the latter hasn't been implemented yet is that it requires changes to the compiler. So, I don't see any reason to make those a library solution. If we're talking something massively more complicated than that, then yes, a library solution starts making more sense. I also think that it then doesn't necessarily make sense to put it in the standard library. If the issue is how the tests print out on failure, that could probably be done in the language as well, depending on what you were talking about changing. If you want to do something massively complicated, however, then you'd probably have to come up with a library solution, but again, I see no reason to have it in the standard library if you're doing anything fancy with how the test failures print out. - Jonathan M Davis
Jan 01 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, January 01, 2012 15:32:34 Jacob Carlborg wrote:
 On 2011-12-31 21:57, Jonathan M Davis wrote:
 On Saturday, December 31, 2011 15:48:16 Jacob Carlborg wrote:
 Yes but what happens when there are many failed tests, i.e. may
 AssertErrors that have been thrown? It will just print all after each
 other and you have to count them yourself if you want to know how many
 failed tests there are?

What does the number of failures really matter? You just need to know which ones failed and where. The AssertErrors give you that. - Jonathan M Davis

It can be nice for statistics. And again, how will it be displayed, just all AssertErrors after each other?

Yes. - Jonathan M Davis
Jan 01 2012
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, January 01, 2012 15:31:18 Jacob Carlborg wrote:
 What's wrong with being able to run the unit tests from your editor,
 have the unit test framework output HTML (or similar), displayed in your
 editor and then you can click on links in the stack trace to get to the
 source code. If you don't see why that's useful that we can just end
 this discussion now.

If you want fancier unit test facilities in your own code. Fine. I just don't think that they should be in the standard library. I think that on the whole, D's built-in unit test framework works fine as it is (barring a few tweaks such as making it so that all unittest blocks within a module run), and that further additions are a needless complication to the standard unit testing facilities and better left to 3rd party solutions where you can do whatever you want with them. - Jonathan M Davis
Jan 01 2012