www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DustMite: the General-Purpose Data Reduction Tool (from the D Blog)

reply Mike Parker <aldacron gmail.com> writes:
Vladimir has contributed to the blog an article on the evolution 
of DustMite, looking at some of the challenges he had to overcome 
along the way.

The blog:
https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/

Reddit:
https://www.reddit.com/r/programming/comments/g0ihse/dustmite_the_generalpurpose_data_reduction_tool/
Apr 13
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Monday, 13 April 2020 at 13:06:30 UTC, Mike Parker wrote:
 Vladimir has contributed to the blog an article on the 
 evolution of DustMite, looking at some of the challenges he had 
 to overcome along the way.

 The blog:
 https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/

 Reddit:
 https://www.reddit.com/r/programming/comments/g0ihse/dustmite_the_generalpurpose_data_reduction_tool/
HN: https://news.ycombinator.com/item?id=22855633
Apr 13
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Monday, 13 April 2020 at 13:06:30 UTC, Mike Parker wrote:
 Vladimir has contributed to the blog an article on the 
 evolution of DustMite, looking at some of the challenges he had 
 to overcome along the way.

 The blog:
 https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/

 Reddit:
 https://www.reddit.com/r/programming/comments/g0ihse/dustmite_the_generalpurpose_data_reduction_tool/
I wish I had even half (or quarter?) of the ingenuity of Vladimir. Really great write-up. Dustmite helped me reduce many compiler bugs in the past and saved me a lot of time. Thanks so much for writing Dustmite, and implementing the recent feature I've requested!
Apr 13
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 4/13/20 9:06 AM, Mike Parker wrote:
 Vladimir has contributed to the blog an article on the evolution of 
 DustMite, looking at some of the challenges he had to overcome along the 
 way.
 
 The blog:
 https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-
ata-reduction-tool/ 
 
 
 Reddit:
 https://www.reddit.com/r/programming/comments/g0ihse/dustmite_the_generalpurpose_
ata_reduction_tool/ 
 
 
 
Very nice article! I think everyone can be inspired by the tools that Vladimir writes. Interesting from the animation that it decided that importing std.stdio can be "reduced" to importing std! I see that you can prevent reductions via regex. How do you say "Don't reduce `std\..*` to `std`" or is that possible? In other words, I'm fine with reducing imports, but not that specific reduction. -Steve
Apr 13
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Monday, 13 April 2020 at 18:53:39 UTC, Steven Schveighoffer 
wrote:
 Very nice article!
Thank you!
 Interesting from the animation that it decided that importing 
 std.stdio can be "reduced" to importing std!
Yes, it's a new minor annoyance for all DustMite users :)
 I see that you can prevent reductions via regex.
Regex and similar rules are applied at input parsing time, not on the emitted output.
 How do you say "Don't reduce `std\..*` to `std`" or is that 
 possible? In other words, I'm fine with reducing imports, but 
 not that specific reduction.
The canonical way, right now, is to add something like `if grep -q 'import .*std[;,]' ; then exit 1 ; fi` to the test script. To make this test reusable, it can be saved to e.g. "dustmite-no-std" and DustMite invoked with `dustmite src "dustmite-no-std && ../actual-test-script.sh"`. I don't know if it's worth it, but to make this common annoyance easier to handle without baking in more highly-D-specific stuff into a tool which aims to be general-purpose, I'm thinking of the following additions: 1. Allow more than one test command. A reduction is considered successful only if all test commands pass. (It would be the equivalent of chaining them with && in a shell command.) 2. Add built-in tests which can be used in place of a test command, such as ":d-no-std".
Apr 13
prev sibling parent reply WebFreak001 <d.forum webfreak.org> writes:
On Monday, 13 April 2020 at 13:06:30 UTC, Mike Parker wrote:
 Vladimir has contributed to the blog an article on the 
 evolution of DustMite, looking at some of the challenges he had 
 to overcome along the way.

 The blog:
 https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/

 Reddit:
 https://www.reddit.com/r/programming/comments/g0ihse/dustmite_the_generalpurpose_data_reduction_tool/
very nice article! Before I was never sure what I could even use dustmite for and rarely ever used it, but having been shown all these use cases here gives me a lot of ideas for how to potentially use it. I really like all the diagrams and animations in the blog post too, they make it a lot more intuitive to grasp what was being done. Though I think some diagrams could have used a little more labels on what the colors, shapes and numbers mean. Also for the performance changes: what do the numbers mean in the diagram there? Is higher better? What exactly is the unit of these numbers? Should I even read it from top to bottom or from bottom to top like usual git logs? Why did it jump from 487 to 200 and is that good or bad?
Apr 14
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Tuesday, 14 April 2020 at 07:03:42 UTC, WebFreak001 wrote:
 very nice article!
Thank you!
 Also for the performance changes: what do the numbers mean in 
 the diagram there? Is higher better? What exactly is the unit 
 of these numbers? Should I even read it from top to bottom or 
 from bottom to top like usual git logs? Why did it jump from 
 487 to 200 and is that good or bad?
Yes, sorry, I was going to add some explanations but forgot. They go in chronological order and going from 200 to 487 was bad (the new tree data structure added a lot of overhead, which was not yet taken advantage of). The numbers represent seconds for 200 reduction steps for reducing my test program. The source code for all programs/scripts used to create the graphics can be found here: https://gitlab.com/CyberShadow/dustmite-article
Apr 15
parent Walter Bright <newshound2 digitalmars.com> writes:
Please do an AMA on the Reddit article!
Apr 15