www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - silly is released - new test runner for the D programming language

reply Anton Fediushin <fediushin.anton yandex.com> writes:
Hello, I'm glad to announce that silly v0.0.1 is released.

Silly is a brand-new test runner with simplicity in mind. It's 
developed to be as simple as possible and contain no useless 
features. Another important goal is to provide flexible tool 
which can be easily integrated into existing environments.

Instead of using dub as part of the test runner or hacking into 
`preBuildCommands` silly seamlessly integrates into `dub test` 
requiring user to just add it as a dependency into dub.json or 
dub.sdl.

Check out project's website [1], repository [2] and page on dub 
registry [3] where you can find more information.

[1]: https://ohboi.gitlab.io/silly/
[2]: https://gitlab.com/ohboi/silly
[3]: https://silly.dub.pm/

Cheers,
Anton
Aug 12
next sibling parent reply Dechcaudron <no-reply no-email.com> writes:
On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:
 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.
I like the "add as dependency and you're done" thing (unfortunately you have to apply conditional compilation to main(). But quick question (just playing the devil's advocate). Why use this library instead of, say, unit-threaded?
Aug 12
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:
 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.
I like the "add as dependency and you're done" thing (unfortunately you have to apply conditional compilation to main(). But quick question (just playing the devil's advocate). Why use this library instead of, say, unit-threaded?
Problem with unit-threaded and similar tools is that they are too complicated for no particular reason. Hacking into dub.json to add some scripting into it is not something everybody wants to waste their time on. Trial might be the first test runner to solve this problem because it just runs as a stand-alone tool and uses dub internally, but it takes forever to compile. It uses too many compile-time features which results in high RAM usage and slow compilation times. Another thing, these tools are trying to be everything people might need adding all kinds of features nobody really uses. For example, assertions in unit-threaded and a lot of different reporters in trial. These tools also advertise usage of built-in `unittest` blocks for integration testing. I think it's just wrong because `unittest`s are obviously meant for unit testing and slapping integration tests on with some duct tape and zip ties is not a good solution. Integration testing needs it's own tool and it's quite possible that I'll end up writing one soon or later. Silly is just my attempt to improve current state of D's ecosystem where programmers don't use advanced test runners, well, because it doesn't worth it for small projects.
Aug 12
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin wrote:
 On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:
 Problem with unit-threaded and similar tools is that they are 
 too complicated for no particular reason.
Yes, I often wake up and think to myself "what feature can I add that aren't going to be useful to anyone at all?" then write some code. Nobody adds complication "for no particular reason". Even people who shouldn't be doing it do it because they're trying to accomplish something else.
 Hacking into dub.json to add some scripting into it is not 
 something everybody wants to waste their time on.
Agreed. Other than manually listing modules however (which is what I do these days anyway due to a dub bug), I don't know of a better way. I wish there were a compile-time trait like __traits(allModules). I've thought of writing a DSI
 Another thing, these tools are trying to be everything people 
 might need adding all kinds of features nobody really uses. For 
 example, assertions in unit-threaded and a lot of different 
 reporters in trial.
This is a good point. I think I should split unit-threaded up into separate dub subpackages. Thanks for the idea!
 These tools also advertise usage of built-in `unittest` blocks 
 for integration testing. I think it's just wrong because 
 `unittest`s are obviously meant for unit testing and slapping 
 integration tests on with some duct tape and zip ties is not a 
 good solution. Integration testing needs it's own tool and it's 
 quite possible that I'll end up writing one soon or later.
I'm curious as to why you think that's the case.
 Silly is just my attempt to improve current state of D's 
 ecosystem where programmers don't use advanced test runners, 
 well, because it doesn't worth it for small projects.
The trouble with small projects is that they tend to not remain small if people find them useful. Also, even if they do, I'd still rather use a good test runner, the same way I don't think any project is too small for git.
Aug 13
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Monday, 13 August 2018 at 11:57:34 UTC, Atila Neves wrote:
 On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin 
 wrote:
 On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:
 Problem with unit-threaded and similar tools is that they are 
 too complicated for no particular reason.
Yes, I often wake up and think to myself "what feature can I add that aren't going to be useful to anyone at all?" then write some code. Nobody adds complication "for no particular reason". Even people who shouldn't be doing it do it because they're trying to accomplish something else.
 Hacking into dub.json to add some scripting into it is not 
 something everybody wants to waste their time on.
Agreed. Other than manually listing modules however (which is what I do these days anyway due to a dub bug), I don't know of a better way. I wish there were a compile-time trait like __traits(allModules). I've thought of writing a DSI
Sadly, there's no better way yet. Silly imports `dub_test_root` which contains all of the modules. That's why no `main()` should be defined and tests must be run with `dub test`.
 Another thing, these tools are trying to be everything people 
 might need adding all kinds of features nobody really uses. 
 For example, assertions in unit-threaded and a lot of 
 different reporters in trial.
This is a good point. I think I should split unit-threaded up into separate dub subpackages. Thanks for the idea!
That'd be nice.
 These tools also advertise usage of built-in `unittest` blocks 
 for integration testing. I think it's just wrong because 
 `unittest`s are obviously meant for unit testing and slapping 
 integration tests on with some duct tape and zip ties is not a 
 good solution. Integration testing needs it's own tool and 
 it's quite possible that I'll end up writing one soon or later.
I'm curious as to why you think that's the case.
Well, 'advertise' might not be the right word for that, but both unit-threaded and trial provide means to write integration tests. Even though there is support for some simple things like creating temporary files and defining steps, it goes beyond "unit testing" but is not enough for integration tests. This kind of testing can require things like setting up a database or running other services which cannot be done with existing tools.
 ...
Aug 13
parent reply Bogdan Szabo <szabobogdan3 gmail.com> writes:
Nice work Anton!

It's nice to see that people are interested in writing better 
test runners. This project reminds me of `tested`: 
http://code.dlang.org/packages/tested


On Monday, 13 August 2018 at 17:32:32 UTC, Anton Fediushin wrote:
 On Monday, 13 August 2018 at 11:57:34 UTC, Atila Neves wrote:
 On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin 
 wrote:
 On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:
 Problem with unit-threaded and similar tools is that they are 
 too complicated for no particular reason.
Yes, I often wake up and think to myself "what feature can I add that aren't going to be useful to anyone at all?" then write some code. Nobody adds complication "for no particular reason". Even people who shouldn't be doing it do it because they're trying to accomplish something else.
Yes... and the main problem is that usually the features added fit the author vision and that vision it's hard to fit all the project.
 Hacking into dub.json to add some scripting into it is not 
 something everybody wants to waste their time on.
Agreed. Other than manually listing modules however (which is what I do these days anyway due to a dub bug), I don't know of a better way. I wish there were a compile-time trait like __traits(allModules). I've thought of writing a DSI
Sadly, there's no better way yet. Silly imports `dub_test_root` which contains all of the modules. That's why no `main()` should be defined and tests must be run with `dub test
embedding the runner code in a project is tricky for compiled programming language... maybe that's why javascript and ruby have a good selection of test runners. I tried to hide the mess in the runner subpackage... I'm not proud of it, but it works.
 Another thing, these tools are trying to be everything people 
 might need adding all kinds of features nobody really uses. 
 For example, assertions in unit-threaded and a lot of 
 different reporters in trial.
This is a good point. I think I should split unit-threaded up into separate dub subpackages. Thanks for the idea!
That'd be nice.
The reporters from trial are optional, and you can tell the runner to use whatever you want. I insisted on the reporters part because I think this is an important feature of a test runner. I can't think of how you would manage thousands of tests without good reporting.
 These tools also advertise usage of built-in `unittest` 
 blocks for integration testing. I think it's just wrong 
 because `unittest`s are obviously meant for unit testing and 
 slapping integration tests on with some duct tape and zip 
 ties is not a good solution. Integration testing needs it's 
 own tool and it's quite possible that I'll end up writing one 
 soon or later.
I'm curious as to why you think that's the case.
Well, 'advertise' might not be the right word for that, but both unit-threaded and trial provide means to write integration tests. Even though there is support for some simple things like creating temporary files and defining steps, it goes beyond "unit testing" but is not enough for integration tests. This kind of testing can require things like setting up a database or running other services which cannot be done with existing tools.
 ...
For me, integration tests are long and sometimes are too many. When you embed the integration tests inside the source code makes harder to find the tested code. I know that we lack of features for the integration tests, but my time is limited and I would like work on other projects too... I would like to see if we can make those runners compatible, in the sense that if someone wants to switch from silly to unit-threaded or trial, to be done with no headaches.
Aug 15
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 15 August 2018 at 08:42:46 UTC, Bogdan Szabo wrote:
 Nice work Anton!

 It's nice to see that people are interested in writing better 
 test runners. This project reminds me of `tested`: 
 http://code.dlang.org/packages/tested
Thanks, tested works just like unit-threaded - list of modules has to be passed to a special template. Silly, on the other hand, exploits features of dub to do its job. <snip>
 Well, 'advertise' might not be the right word for that, but 
 both unit-threaded and trial provide means to write 
 integration tests. Even though there is support for some 
 simple things like creating temporary files and defining 
 steps, it goes beyond "unit testing" but is not enough for 
 integration tests. This kind of testing can require things 
 like setting up a database or running other services which 
 cannot be done with existing tools.

 ...
For me, integration tests are long and sometimes are too many. When you embed the integration tests inside the source code makes harder to find the tested code. I know that we lack of features for the integration tests, but my time is limited and I would like work on other projects too...
It's ok, we're all enthusiasts here.
 I would like to see if we can make those runners compatible, in 
 the sense that if someone wants to switch from silly to 
 unit-threaded or trial, to be done with no headaches.
Removing silly will be simple enough, but switching between runners... not so much, it'd require different `configurations` in dub.json for different runners. What we could do though is improve dub to handle this in the same way it uses tested to run tests when it's installed [1]. [1]: https://github.com/dlang/dub/blob/9914720ca5dcd9b37eb2ec0ffb08f218642a1960/source/dub/dub.d#L717
Aug 15
parent reply Bogdan Szabo <szabobogdan3 gmail.com> writes:
On Wednesday, 15 August 2018 at 11:26:20 UTC, Anton Fediushin 
wrote:
 On Wednesday, 15 August 2018 at 08:42:46 UTC, Bogdan Szabo 
 wrote:
 Nice work Anton!

 It's nice to see that people are interested in writing better 
 test runners. This project reminds me of `tested`: 
 http://code.dlang.org/packages/tested
Thanks, tested works just like unit-threaded - list of modules has to be passed to a special template. Silly, on the other hand, exploits features of dub to do its job. <snip>
 Well, 'advertise' might not be the right word for that, but 
 both unit-threaded and trial provide means to write 
 integration tests. Even though there is support for some 
 simple things like creating temporary files and defining 
 steps, it goes beyond "unit testing" but is not enough for 
 integration tests. This kind of testing can require things 
 like setting up a database or running other services which 
 cannot be done with existing tools.

 ...
For me, integration tests are long and sometimes are too many. When you embed the integration tests inside the source code makes harder to find the tested code. I know that we lack of features for the integration tests, but my time is limited and I would like work on other projects too...
It's ok, we're all enthusiasts here.
 I would like to see if we can make those runners compatible, 
 in the sense that if someone wants to switch from silly to 
 unit-threaded or trial, to be done with no headaches.
Removing silly will be simple enough, but switching between runners... not so much, it'd require different `configurations` in dub.json for different runners. What we could do though is improve dub to handle this in the same way it uses tested to run tests when it's installed [1]. [1]: https://github.com/dlang/dub/blob/9914720ca5dcd9b37eb2ec0ffb08f218642a1960/source/dub/dub.d#L717
I wonder if the test runners could provide a template for the injected code, that dub could use it on `dub test`. I wonder if we could add something like `"testRunner": "trial"` in the dub.json file.
Aug 15
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 16/08/2018 1:31 AM, Bogdan Szabo wrote:
 I wonder if the test runners could provide a template for the injected 
 code, that dub could use it on `dub test`. I wonder if we could add 
 something like `"testRunner": "trial"` in the dub.json file.
I've thought about this a bit. Adding special support for test runners does seem like a bad idea. An alternative could be to enable injection of a mixin template by the compiler for each compiler parsed. You can either get an alias to initialize a template for only type purposes, or you can throw a module constructor in. That would enable more interesting use cases like route registration for web routers.
Aug 15
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 15 August 2018 at 14:01:24 UTC, rikki cattermole 
wrote:
 On 16/08/2018 1:31 AM, Bogdan Szabo wrote:
 I wonder if the test runners could provide a template for the 
 injected code, that dub could use it on `dub test`. I wonder 
 if we could add something like `"testRunner": "trial"` in the 
 dub.json file.
I've thought about this a bit. Adding special support for test runners does seem like a bad idea. An alternative could be to enable injection of a mixin template by the compiler for each compiler parsed. You can either get an alias to initialize a template for only type purposes, or you can throw a module constructor in. That would enable more interesting use cases like route registration for web routers.
I disagree, adding special support for test runners is great, in fact dub already has that (it uses `tested` if it's installed). What we need to do is just extend this feature a little bit and tell the ones who develop test runners to update them accordingly. I'd be happy to have that and I think Atila and Bogdan too. Also, dub has special support for documentation generators that are compatible with ddox. There's `"-ddoxTool"` configuration option. I think we could do the same with test runners being compatible with tested.
Aug 15
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/18 11:07 AM, Anton Fediushin wrote:
 Hello, I'm glad to announce that silly v0.0.1 is released.
 
 Silly is a brand-new test runner with simplicity in mind. It's developed 
 to be as simple as possible and contain no useless features. Another 
 important goal is to provide flexible tool which can be easily 
 integrated into existing environments.
 
 Instead of using dub as part of the test runner or hacking into 
 `preBuildCommands` silly seamlessly integrates into `dub test` requiring 
 user to just add it as a dependency into dub.json or dub.sdl.
 
 Check out project's website [1], repository [2] and page on dub registry 
 [3] where you can find more information.
 
 [1]: https://ohboi.gitlab.io/silly/
 [2]: https://gitlab.com/ohboi/silly
 [3]: https://silly.dub.pm/
 
I notice that you're using the new unittest system added in 2.078, but bypassing the reporting mechanism (i.e. using exit instead of returning). Is there a reason why? This is exactly the type of thing I envisioned would be used when developing said feature. I'm asking because I'd like to understand what the limitations are that caused you to bypass it. -Steve
Aug 15
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 15 August 2018 at 14:33:26 UTC, Steven 
Schveighoffer wrote:
 On 8/12/18 11:07 AM, Anton Fediushin wrote:
 Hello, I'm glad to announce that silly v0.0.1 is released.
 
 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.
 
 Instead of using dub as part of the test runner or hacking 
 into `preBuildCommands` silly seamlessly integrates into `dub 
 test` requiring user to just add it as a dependency into 
 dub.json or dub.sdl.
 
 Check out project's website [1], repository [2] and page on 
 dub registry [3] where you can find more information.
 
 [1]: https://ohboi.gitlab.io/silly/
 [2]: https://gitlab.com/ohboi/silly
 [3]: https://silly.dub.pm/
 
I notice that you're using the new unittest system added in 2.078, but bypassing the reporting mechanism (i.e. using exit instead of returning). Is there a reason why? This is exactly the type of thing I envisioned would be used when developing said feature. I'm asking because I'd like to understand what the limitations are that caused you to bypass it. -Steve
Thanks for pointing it out, I never really looked into that. I used `exit` before because everything was split into functions, so it was simpler to just `exit` than return things here and there. It's fixed now in v0.0.2. I'm still using own printer for summary, because built-in one is boring: --- Summary: 2 passed, 2 failed in 0 ms --- vs --- 2/4 unittests FAILED ---
Aug 15
prev sibling parent reply Soulsbane <viserion.thrall gmail.com> writes:
On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:
 Hello, I'm glad to announce that silly v0.0.1 is released.

 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.
In my local version I've modified the test name to be colorized. Could you add this feature? it really helps the readability in my opinion. Either way, I really love silly! Thanks a lot!
Sep 11
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 12 September 2018 at 04:02:14 UTC, Soulsbane wrote:
 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:
 Hello, I'm glad to announce that silly v0.0.1 is released.

 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.
In my local version I've modified the test name to be colorized. Could you add this feature? it really helps the readability in my opinion.
I doubt that will make it more readable, in fact it'd annoy me. Another thing to consider is the fact that colours in terminal are highly customizable so if something works and most importantly looks good for somebody it might look terrible and be unreadable on different terminal preferences. This is something I experienced with trial which colours test names white and vibe-core's logger which used different shades of grey for different log levels. Both of these as you can imagine are unreadable on black-on-white terminals.
 Either way, I really love silly! Thanks a lot!
You are welcome!
Sep 12