www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - silly is released - new test runner for the D programming language

reply Anton Fediushin <fediushin.anton yandex.com> writes:
Hello, I'm glad to announce that silly v0.0.1 is released.

Silly is a brand-new test runner with simplicity in mind. It's 
developed to be as simple as possible and contain no useless 
features. Another important goal is to provide flexible tool 
which can be easily integrated into existing environments.

Instead of using dub as part of the test runner or hacking into 
`preBuildCommands` silly seamlessly integrates into `dub test` 
requiring user to just add it as a dependency into dub.json or 
dub.sdl.

Check out project's website [1], repository [2] and page on dub 
registry [3] where you can find more information.

[1]: https://ohboi.gitlab.io/silly/
[2]: https://gitlab.com/ohboi/silly
[3]: https://silly.dub.pm/

Cheers,
Anton
Aug 12 2018
next sibling parent reply Dechcaudron <no-reply no-email.com> writes:
On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:

 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.


I like the "add as dependency and you're done" thing 
(unfortunately you have to apply conditional compilation to 
main().

But quick question (just playing the devil's advocate). Why use 
this library instead of, say, unit-threaded?
Aug 12 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:

 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:

 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.


 I like the "add as dependency and you're done" thing 
 (unfortunately you have to apply conditional compilation to 
 main().

 But quick question (just playing the devil's advocate). Why use 
 this library instead of, say, unit-threaded?


Problem with unit-threaded and similar tools is that they are too 
complicated for no particular reason. Hacking into dub.json to 
add some scripting into it is not something everybody wants to 
waste their time on.

Trial might be the first test runner to solve this problem 
because it just runs as a stand-alone tool and uses dub 
internally, but it takes forever to compile. It uses too many 
compile-time features which results in high RAM usage and slow 
compilation times.

Another thing, these tools are trying to be everything people 
might need adding all kinds of features nobody really uses. For 
example, assertions in unit-threaded and a lot of different 
reporters in trial.

These tools also advertise usage of built-in `unittest` blocks 
for integration testing. I think it's just wrong because 
`unittest`s are obviously meant for unit testing and slapping 
integration tests on with some duct tape and zip ties is not a 
good solution. Integration testing needs it's own tool and it's 
quite possible that I'll end up writing one soon or later.

Silly is just my attempt to improve current state of D's 
ecosystem where programmers don't use advanced test runners, 
well, because it doesn't worth it for small projects.
Aug 12 2018
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin wrote:

 On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:

 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:





 Problem with unit-threaded and similar tools is that they are 
 too complicated for no particular reason.


Yes, I often wake up and think to myself "what feature can I add 
that aren't going to be useful to anyone at all?" then write some 
code.

Nobody adds complication "for no particular reason". Even people 
who shouldn't be doing it do it because they're trying to 
accomplish something else.


 Hacking into dub.json to add some scripting into it is not 
 something everybody wants to waste their time on.


Agreed. Other than manually listing modules however (which is 
what I do these days anyway due to a dub bug), I don't know of a 
better way.

I wish there were a compile-time trait like __traits(allModules). 
I've thought of writing a DSI


 Another thing, these tools are trying to be everything people 
 might need adding all kinds of features nobody really uses. For 
 example, assertions in unit-threaded and a lot of different 
 reporters in trial.


This is a good point. I think I should split unit-threaded up 
into separate dub subpackages. Thanks for the idea!


 These tools also advertise usage of built-in `unittest` blocks 
 for integration testing. I think it's just wrong because 
 `unittest`s are obviously meant for unit testing and slapping 
 integration tests on with some duct tape and zip ties is not a 
 good solution. Integration testing needs it's own tool and it's 
 quite possible that I'll end up writing one soon or later.


I'm curious as to why you think that's the case.


 Silly is just my attempt to improve current state of D's 
 ecosystem where programmers don't use advanced test runners, 
 well, because it doesn't worth it for small projects.


The trouble with small projects is that they tend to not remain 
small if people find them useful. Also, even if they do, I'd 
still rather use a good test runner, the same way I don't think 
any project is too small for git.
Aug 13 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Monday, 13 August 2018 at 11:57:34 UTC, Atila Neves wrote:

 On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin 
 wrote:

 On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:

 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:





 Problem with unit-threaded and similar tools is that they are 
 too complicated for no particular reason.


 Yes, I often wake up and think to myself "what feature can I 
 add that aren't going to be useful to anyone at all?" then 
 write some code.

 Nobody adds complication "for no particular reason". Even 
 people who shouldn't be doing it do it because they're trying 
 to accomplish something else.


 Hacking into dub.json to add some scripting into it is not 
 something everybody wants to waste their time on.


 Agreed. Other than manually listing modules however (which is 
 what I do these days anyway due to a dub bug), I don't know of 
 a better way.

 I wish there were a compile-time trait like 
 __traits(allModules). I've thought of writing a DSI


Sadly, there's no better way yet. Silly imports `dub_test_root` 
which contains all of the modules. That's why no `main()` should 
be defined and tests must be run with `dub test`.


 Another thing, these tools are trying to be everything people 
 might need adding all kinds of features nobody really uses. 
 For example, assertions in unit-threaded and a lot of 
 different reporters in trial.


 This is a good point. I think I should split unit-threaded up 
 into separate dub subpackages. Thanks for the idea!


That'd be nice.


 These tools also advertise usage of built-in `unittest` blocks 
 for integration testing. I think it's just wrong because 
 `unittest`s are obviously meant for unit testing and slapping 
 integration tests on with some duct tape and zip ties is not a 
 good solution. Integration testing needs it's own tool and 
 it's quite possible that I'll end up writing one soon or later.


 I'm curious as to why you think that's the case.


Well, 'advertise' might not be the right word for that, but both 
unit-threaded and trial provide means to write integration tests. 
Even though there is support for some simple things like creating 
temporary files and defining steps, it goes beyond "unit testing" 
but is not enough for integration tests. This kind of testing can 
require things like setting up a database or running other 
services which cannot be done with existing tools.


 ...
Aug 13 2018
parent reply Bogdan Szabo <szabobogdan3 gmail.com> writes:
Nice work Anton!

It's nice to see that people are interested in writing better 
test runners. This project reminds me of `tested`: 
http://code.dlang.org/packages/tested


On Monday, 13 August 2018 at 17:32:32 UTC, Anton Fediushin wrote:

 On Monday, 13 August 2018 at 11:57:34 UTC, Atila Neves wrote:

 On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin 
 wrote:

 On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:

 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:





 Problem with unit-threaded and similar tools is that they are 
 too complicated for no particular reason.


 Yes, I often wake up and think to myself "what feature can I 
 add that aren't going to be useful to anyone at all?" then 
 write some code.

 Nobody adds complication "for no particular reason". Even 
 people who shouldn't be doing it do it because they're trying 
 to accomplish something else.




Yes... and the main problem is that usually the features added 
fit the author vision and that vision it's hard to fit all the 
project.


 Hacking into dub.json to add some scripting into it is not 
 something everybody wants to waste their time on.


 Agreed. Other than manually listing modules however (which is 
 what I do these days anyway due to a dub bug), I don't know of 
 a better way.

 I wish there were a compile-time trait like 
 __traits(allModules). I've thought of writing a DSI


 Sadly, there's no better way yet. Silly imports `dub_test_root` 
 which contains all of the modules. That's why no `main()` 
 should be defined and tests must be run with `dub test


embedding the runner code in a project is tricky for compiled 
programming language... maybe that's why javascript and ruby have 
a good selection of test runners. I tried to hide the mess in the 
runner subpackage... I'm not proud of it, but it works.


 Another thing, these tools are trying to be everything people 
 might need adding all kinds of features nobody really uses. 
 For example, assertions in unit-threaded and a lot of 
 different reporters in trial.


 This is a good point. I think I should split unit-threaded up 
 into separate dub subpackages. Thanks for the idea!


 That'd be nice.


The reporters from trial are optional, and you can tell the 
runner to use whatever you want. I insisted on the reporters part 
because I think this is an important feature of a test runner. I 
can't think of how you would manage thousands of tests without 
good reporting.


 These tools also advertise usage of built-in `unittest` 
 blocks for integration testing. I think it's just wrong 
 because `unittest`s are obviously meant for unit testing and 
 slapping integration tests on with some duct tape and zip 
 ties is not a good solution. Integration testing needs it's 
 own tool and it's quite possible that I'll end up writing one 
 soon or later.


 I'm curious as to why you think that's the case.


 Well, 'advertise' might not be the right word for that, but 
 both unit-threaded and trial provide means to write integration 
 tests. Even though there is support for some simple things like 
 creating temporary files and defining steps, it goes beyond 
 "unit testing" but is not enough for integration tests. This 
 kind of testing can require things like setting up a database 
 or running other services which cannot be done with existing 
 tools.


 ...




For me, integration tests are long and sometimes are too many. 
When you embed the integration tests inside the source code makes 
harder to find the tested code.

I know that we lack of features for the integration tests, but my 
time is limited and I would like work on other projects too...

I would like to see if we can make those runners compatible, in 
the sense that if someone wants to switch from silly to 
unit-threaded or trial, to be done with no headaches.
Aug 15 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 15 August 2018 at 08:42:46 UTC, Bogdan Szabo wrote:

 Nice work Anton!

 It's nice to see that people are interested in writing better 
 test runners. This project reminds me of `tested`: 
 http://code.dlang.org/packages/tested


Thanks, tested works just like unit-threaded - list of modules 
has to be passed to a special template. Silly, on the other hand, 
exploits features of dub to do its job.

<snip>


 Well, 'advertise' might not be the right word for that, but 
 both unit-threaded and trial provide means to write 
 integration tests. Even though there is support for some 
 simple things like creating temporary files and defining 
 steps, it goes beyond "unit testing" but is not enough for 
 integration tests. This kind of testing can require things 
 like setting up a database or running other services which 
 cannot be done with existing tools.


 ...




 For me, integration tests are long and sometimes are too many. 
 When you embed the integration tests inside the source code 
 makes harder to find the tested code.

 I know that we lack of features for the integration tests, but 
 my time is limited and I would like work on other projects 
 too...


It's ok, we're all enthusiasts here.


 I would like to see if we can make those runners compatible, in 
 the sense that if someone wants to switch from silly to 
 unit-threaded or trial, to be done with no headaches.


Removing silly will be simple enough, but switching between 
runners... not so much, it'd require different `configurations` 
in dub.json for different runners.

What we could do though is improve dub to handle this in the same 
way it uses tested to run tests when it's installed [1].

[1]: 
https://github.com/dlang/dub/blob/9914720ca5dcd9b37eb2ec0ffb08f218642a1960/source/dub/dub.d#L717
Aug 15 2018
parent reply Bogdan Szabo <szabobogdan3 gmail.com> writes:
On Wednesday, 15 August 2018 at 11:26:20 UTC, Anton Fediushin 
wrote:

 On Wednesday, 15 August 2018 at 08:42:46 UTC, Bogdan Szabo 
 wrote:

 Nice work Anton!

 It's nice to see that people are interested in writing better 
 test runners. This project reminds me of `tested`: 
 http://code.dlang.org/packages/tested


 Thanks, tested works just like unit-threaded - list of modules 
 has to be passed to a special template. Silly, on the other 
 hand, exploits features of dub to do its job.

 <snip>


 Well, 'advertise' might not be the right word for that, but 
 both unit-threaded and trial provide means to write 
 integration tests. Even though there is support for some 
 simple things like creating temporary files and defining 
 steps, it goes beyond "unit testing" but is not enough for 
 integration tests. This kind of testing can require things 
 like setting up a database or running other services which 
 cannot be done with existing tools.


 ...




 For me, integration tests are long and sometimes are too many. 
 When you embed the integration tests inside the source code 
 makes harder to find the tested code.

 I know that we lack of features for the integration tests, but 
 my time is limited and I would like work on other projects 
 too...


 It's ok, we're all enthusiasts here.


 I would like to see if we can make those runners compatible, 
 in the sense that if someone wants to switch from silly to 
 unit-threaded or trial, to be done with no headaches.


 Removing silly will be simple enough, but switching between 
 runners... not so much, it'd require different `configurations` 
 in dub.json for different runners.

 What we could do though is improve dub to handle this in the 
 same way it uses tested to run tests when it's installed [1].

 [1]: 
 https://github.com/dlang/dub/blob/9914720ca5dcd9b37eb2ec0ffb08f218642a1960/source/dub/dub.d#L717


I wonder if the test runners could provide a template for the 
injected code, that dub could use it on `dub test`. I wonder if 
we could add something like `"testRunner": "trial"` in the 
dub.json file.
Aug 15 2018
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 16/08/2018 1:31 AM, Bogdan Szabo wrote:

 I wonder if the test runners could provide a template for the injected 
 code, that dub could use it on `dub test`. I wonder if we could add 
 something like `"testRunner": "trial"` in the dub.json file.


I've thought about this a bit. Adding special support for test runners 
does seem like a bad idea. An alternative could be to enable injection 
of a mixin template by the compiler for each compiler parsed. You can 
either get an alias to initialize a template for only type purposes, or 
you can throw a module constructor in.

That would enable more interesting use cases like route registration for 
web routers.
Aug 15 2018
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 15 August 2018 at 14:01:24 UTC, rikki cattermole 
wrote:

 On 16/08/2018 1:31 AM, Bogdan Szabo wrote:

 I wonder if the test runners could provide a template for the 
 injected code, that dub could use it on `dub test`. I wonder 
 if we could add something like `"testRunner": "trial"` in the 
 dub.json file.


 I've thought about this a bit. Adding special support for test 
 runners does seem like a bad idea. An alternative could be to 
 enable injection of a mixin template by the compiler for each 
 compiler parsed. You can either get an alias to initialize a 
 template for only type purposes, or you can throw a module 
 constructor in.

 That would enable more interesting use cases like route 
 registration for web routers.


I disagree, adding special support for test runners is great, in 
fact dub already has that (it uses `tested` if it's installed). 
What we need to do is just extend this feature a little bit and 
tell the ones who develop test runners to update them 
accordingly. I'd be happy to have that and I think Atila and 
Bogdan too.

Also, dub has special support for documentation generators that 
are compatible with ddox. There's `"-ddoxTool"` configuration 
option. I think we could do the same with test runners being 
compatible with tested.
Aug 15 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/18 11:07 AM, Anton Fediushin wrote:

 Hello, I'm glad to announce that silly v0.0.1 is released.
 
 Silly is a brand-new test runner with simplicity in mind. It's developed 
 to be as simple as possible and contain no useless features. Another 
 important goal is to provide flexible tool which can be easily 
 integrated into existing environments.
 
 Instead of using dub as part of the test runner or hacking into 
 `preBuildCommands` silly seamlessly integrates into `dub test` requiring 
 user to just add it as a dependency into dub.json or dub.sdl.
 
 Check out project's website [1], repository [2] and page on dub registry 
 [3] where you can find more information.
 
 [1]: https://ohboi.gitlab.io/silly/
 [2]: https://gitlab.com/ohboi/silly
 [3]: https://silly.dub.pm/
 


I notice that you're using the new unittest system added in 2.078, but 
bypassing the reporting mechanism (i.e. using exit instead of 
returning). Is there a reason why?

This is exactly the type of thing I envisioned would be used when 
developing said feature. I'm asking because I'd like to understand what 
the limitations are that caused you to bypass it.

-Steve
Aug 15 2018
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 15 August 2018 at 14:33:26 UTC, Steven 
Schveighoffer wrote:

 On 8/12/18 11:07 AM, Anton Fediushin wrote:

 Hello, I'm glad to announce that silly v0.0.1 is released.
 
 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.
 
 Instead of using dub as part of the test runner or hacking 
 into `preBuildCommands` silly seamlessly integrates into `dub 
 test` requiring user to just add it as a dependency into 
 dub.json or dub.sdl.
 
 Check out project's website [1], repository [2] and page on 
 dub registry [3] where you can find more information.
 
 [1]: https://ohboi.gitlab.io/silly/
 [2]: https://gitlab.com/ohboi/silly
 [3]: https://silly.dub.pm/
 


 I notice that you're using the new unittest system added in 
 2.078, but bypassing the reporting mechanism (i.e. using exit 
 instead of returning). Is there a reason why?

 This is exactly the type of thing I envisioned would be used 
 when developing said feature. I'm asking because I'd like to 
 understand what the limitations are that caused you to bypass 
 it.

 -Steve


Thanks for pointing it out, I never really looked into that. I 
used `exit` before because everything was split into functions, 
so it was simpler to just `exit` than return things here and 
there.

It's fixed now in v0.0.2. I'm still using own printer for 
summary, because built-in one is boring:

---
Summary: 2 passed, 2 failed in 0 ms
---
vs
---
2/4 unittests FAILED
---
Aug 15 2018
prev sibling parent reply Soulsbane <viserion.thrall gmail.com> writes:
On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin wrote:

 Hello, I'm glad to announce that silly v0.0.1 is released.

 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.


In my local version I've modified the test name to be colorized. 
Could you add this feature? it really helps the readability in my 
opinion.

Either way, I really love silly! Thanks a lot!
Sep 11 2018
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Wednesday, 12 September 2018 at 04:02:14 UTC, Soulsbane wrote:

 On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
 wrote:

 Hello, I'm glad to announce that silly v0.0.1 is released.

 Silly is a brand-new test runner with simplicity in mind. It's 
 developed to be as simple as possible and contain no useless 
 features. Another important goal is to provide flexible tool 
 which can be easily integrated into existing environments.


 In my local version I've modified the test name to be 
 colorized. Could you add this feature? it really helps the 
 readability in my opinion.


I doubt that will make it more readable, in fact it'd annoy me. 
Another thing to consider is the fact that colours in terminal 
are highly customizable so if something works and most 
importantly looks good for somebody it might look terrible and be 
unreadable on different terminal preferences.

This is something I experienced with trial which colours test 
names white and vibe-core's logger which used different shades of 
grey for different log levels. Both of these as you can imagine 
are unreadable on black-on-white terminals.


 Either way, I really love silly! Thanks a lot!


You are welcome!
Sep 12 2018