www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - We're long overdue for a "D is awesome" post

reply "H. S. Teoh" <hsteoh qfbox.info> writes:
So here's one.

Today, I was writing some code that iterates over a data structure and
writes output to bunch of different files. It looks something like this:

void genSplitHtml(Data data, ...) {
	auto outputTemplate = File("template.html", "r");
	foreach (...) {
		auto filename = generateFilename(...);
		auto sink = File(filename, "w").lockingTextWriter;
		...
		while (line; outputTemplate.byLine) {
			if (line.canFind("MAGIC_TOKEN")) {
				generateOutput(...);
			} else {
				sink.put(line);
			}
		}
	}
}

Since the whole point of this function was to write output to different
files (with automatically determined names), I wanted to write a
unittest that tests whether it creates the correct files with the
correct contents.  But I didn't want unittests to touch the actual
filesystem either -- didn't want to have to clean up the mess during
development where the code might sometimes break and leave detritus
behind in a temporary directory, or interact badly with other unittests
running in parallel, etc..

And I didn't want to do a massive code refactoring to make genSplitHtml
more unittestable. (Which would have more chances of screwing up and
having bugs creep in, which defeats the purpose of this exercise.)

So I came up with this solution:

1) Rewrite the function to:

	void genSplitHtml(File = std.stdio.File)(Data data, ...) { ... }

   The default parameter binds to the real filesystem by default, so
   other code that uses this function don't have to change to deal with
   a new API.  Then for the unittest code:

2) Create a fake virtual filesystem in my unittest block:

	static struct MockFile {
		static string[string] files;

		string fname;
		this(string _fname, string mode) {
			// ignore `mode` for this test
			fname = _fname;
		}

		// Mock replacement for std.stdio.File.lockingTextWriter
		auto lockingTextWriter() {
			return (const(char)[] data) {
				// simulate writing to a file
				files[fname] ~= data.dup;
			};
		}
		void rewind() {} // dummy
		void close() {} // dummy
		auto byLine() {
			// We're mostly writing to files, and only
			// reading from a specific one. So just fake its
			// contents here.
			if (fname != "template.html") return [];
			else return [
				"<html>",
				"MAGIC_TOKEN",
				"</html>"
			];
		}
	}

Then instead of calling genSplitHtml(...), the unittest calls it as
genSplitHtml!MockFile(...). This replaces std.stdio.File with MockFile,
and thanks to D templates and the range API, the rest of the code just
adapted itself automatically to the MockFile fake filesystem. After the
function is done, the unittest just has to inspect the contents of
MockFile.files to verify that the correct files are there, and that
their contents are correct.

Took me like 10 minutes to write MockFile and a unittest that checks for
correct behaviour using the fake filesystem.

MockFile can be expanded in the future to simulate, e.g. a full
filesystem, a filesystem that occasionally (or always) fails, or
corrupts data, etc.: test cases that would be impractical to test with a
real filesystem.  Best of all, I get all of this "for free": no existing
code has to change except for that single new template parameter to the
target function.

D not only r0x0rs, D boulders!!


T

-- 
Heads I win, tails you lose.
May 30 2023
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Excellent! The technique you're exhibiting is called "mocking". It works 
especially well when a function's inputs and outputs are properly parameterized.

When I fixed the D lexer.d to not access any globals, I was able to simplify
the 
unittests for it by mocking up the inputs and outputs. For example, instead of 
having to use the global "gagging" switch to suppress error messages, I used a 
mock error handler that just did nothing.


On 5/30/2023 2:42 PM, H. S. Teoh wrote:

 So here's one.
May 30 2023
parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Tue, May 30, 2023 at 04:27:05PM -0700, Walter Bright via Digitalmars-d wrote:

 Excellent! The technique you're exhibiting is called "mocking". It
 works especially well when a function's inputs and outputs are
 properly parameterized.


Yes, it's an age-old technique since at least the days of Java.

What makes it stand out in this particular case, though, is D's ability
to let you mock references to global (module-level) symbols with just a
1-line change.  To mock the top-level `System` in Java, for example, you
have to do some arcane black magic with the help of an external package;
it's not something for mere mortals or the faint of heart to do by hand.
In D, you add a single template parameter to your function, and you're
all set to go.  Now *that's* power.



 When I fixed the D lexer.d to not access any globals, I was able to
 simplify the unittests for it by mocking up the inputs and outputs.
 For example, instead of having to use the global "gagging" switch to
 suppress error messages, I used a mock error handler that just did
 nothing.


Good to hear.  Error gagging is the source of some of the most
irritating D compiler bugs in the past; the less it gets used, the
better IMNSHO.


T

-- 
Genius may have its limitations, but stupidity is not thus handicapped. --
Elbert Hubbard
May 30 2023
prev sibling next sibling parent reply Salih Dincer <salihdb hotmail.com> writes:
On Tuesday, 30 May 2023 at 21:42:04 UTC, H. S. Teoh wrote:

 ```d
 void genSplitHtml(Data data, ...) {
 	auto outputTemplate = File("template.html", "r");
 	foreach (...) {
 		auto filename = generateFilename(...);
 		auto sink = File(filename, "w").lockingTextWriter;
 		...
 		while (line; outputTemplate.byLine) {
 			if (line.canFind("MAGIC_TOKEN")) {
 				generateOutput(...);
 			} else {
 				sink.put(line);
 			}
 		}
 	}
 }

 ```


I didn't know that `while` is used just like `foreach`. 0k4y, we 
can use `auto` in `while` condition or `if`, which lives in the 
same block as `auto`.

```d
while (line; outputTemplate.byLine) {...}
```

Does the following really work, thanks?



 ```d
 	static struct MockFile {
 		static string[string] files;

 		string fname;
 		this(string _fname, string mode) {
 			// ignore `mode` for this test
 			fname = _fname;
 		}

 		// Mock replacement for std.stdio.File.lockingTextWriter
 		auto lockingTextWriter() {
 			return (const(char)[] data) {
 				// simulate writing to a file
 				files[fname] ~= data.dup;
 			};
 		}
 		void rewind() {} // dummy
 		void close() {} // dummy
 		auto byLine() {
 			// We're mostly writing to files, and only
 			// reading from a specific one. So just fake its
 			// contents here.
 			if (fname != "template.html") return [];
 			else return [
 				"<html>",
 				"MAGIC_TOKEN",
 				"</html>"
 			];
 		}
 	}
 ```


SDB 79
May 30 2023
parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Wed, May 31, 2023 at 12:28:40AM +0000, Salih Dincer via Digitalmars-d wrote:
[...]

 I didn't know that `while` is used just like `foreach`. 0k4y, we can
 use `auto` in `while` condition or `if`, which lives in the same block
 as `auto`.
 
 ```d
 while (line; outputTemplate.byLine) {...}
 ```
 
 Does the following really work, thanks?


Oops, that's supposed to be `foreach`, not `while`. :-/  My bad.


T

-- 
Trying to define yourself is like trying to bite your own teeth. -- Alan Watts
May 30 2023
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/30/23 5:42 PM, H. S. Teoh wrote:

 So here's one.
 
 Today, I was writing some code that iterates over a data structure and
 writes output to bunch of different files. It looks something like this:
 
 void genSplitHtml(Data data, ...) {
 	auto outputTemplate = File("template.html", "r");
 	foreach (...) {
 		auto filename = generateFilename(...);
 		auto sink = File(filename, "w").lockingTextWriter;
 		...
 		while (line; outputTemplate.byLine) {
 			if (line.canFind("MAGIC_TOKEN")) {
 				generateOutput(...);
 			} else {
 				sink.put(line);
 			}
 		}
 	}
 }



 
 Since the whole point of this function was to write output to different
 files (with automatically determined names), I wanted to write a
 unittest that tests whether it creates the correct files with the
 correct contents.  But I didn't want unittests to touch the actual
 filesystem either -- didn't want to have to clean up the mess during
 development where the code might sometimes break and leave detritus
 behind in a temporary directory, or interact badly with other unittests
 running in parallel, etc..


All of iopipe is 100% unittestable, because all strings are treated as 
input pipes.

Roughly speaking, if this were iopipe, I would do it like:

```d
// returns an iopipe with the translated data
auto genSplitHtml(Input)(Input inputPipe, Data data, ...)
{
    ...
}

// and then you can call it like:

File("template.html", "r").genSplitHtml(data).writeTo(File(filename, 
"w")).process();
// in unittest
auto result = "contents of template".genSplitHtml(data);
result.process(); // oof, would be nice if I could do this in one line
assert(result.window == expectedData);
```

I need to work on it more, that `writeTo` function is not there yet 
(https://github.com/schveiguy/iopipe/issues/39) and the file opening 
stuff isn't as solid as I'd like.


 D not only r0x0rs, D boulders!!


Indeed it does!

-Steve
May 30 2023
parent Salih Dincer <salihdb hotmail.com> writes:
On Wednesday, 31 May 2023 at 00:47:27 UTC, Steven Schveighoffer 
wrote:

 On 5/30/23 5:42 PM, H. S. Teoh wrote:

 D not only r0x0rs, D boulders!!


 Indeed it does!

 -Steve


More than admit it, I want it to look cool and humble at the same 
time. IMO this wil be possible with 
[IVY](https://forum.dlang.org/thread/lktxmzdcmaagtrswpily forum.dlang.org)...

I am working on a project of about 1 Kloc. The project is so 
simple (thanks to D!) that it is unnecessary to explain. However, 
since I am working with string and dchar, I will add JSON and 
File System simultaneously in the future. For now, I don't need 
it at all and I'm not in a rush.

Even a simple wrapper below makes it easy for me to use in a 
foreach. I like to write code with D.

![snippet](http://www.sdb.net.tr/dlang/images/stripRange001.jpg)

SDB 79
May 31 2023
prev sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 5/30/23 14:42, H. S. Teoh wrote:

 So here's one.


Thank you for reminding us. ;)


 	void genSplitHtml(File = std.stdio.File)(Data data, ...) { ... }


Smart! In the past, I used runtime polymorhism to achieve the same. Oops! :)


 MockFile can be expanded in the future to simulate, e.g. a full
 filesystem,


Yep. Mine was using associative arrays keyed by strings (file names) and 
data as dynamic arrays.

Ali
May 31 2023