www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Templates, D way

reply Void-995 <void995 gmail.com> writes:
Hi, everyone. I'm pretty new to D and trying my first steps in 
it. Currently I'm trying to port some code from C/C++ with pretty 
weird data structures and don't like idea of making boilerplate 
functions for accessing sub-lists in main binary structure (lets 
not talk about it's design, it's from legacy thing I want to deal 
with and I can't change the format itself or writing a lot of 
additional parsing code). With hour I spent on trying I've ended 
with what you may see below, but I wonder if there is more 
pleasant variant of re-writing that template as I haven't found 
anything better in either books or online documentation yet:

template DataList(const char[] listName, T, alias dataOffset, 
alias listLength)
{
	const char[] DataList = format(q{
		%s[] %s()
		{
			return (cast(%s *)(cast(byte *)(&this) + %s))[0 .. %s];
		}
	}, T.stringof, listName, T.stringof, dataOffset.stringof, 
listLength.stringof);
}

struct MyBinarySubStructAForA
{
	int someIntegerFieldA;
	float someFloatFieldA;
}

struct MyBinarySubStructBForA
{
	int someIntegerFieldB;
	float someFloatFieldB;
}

struct MyBinaryStructA
{
	int firstSublistMembersCount;
	int firstSublistMembersOffset;

	int secondSublistMembersCount;
	int secondSublistMembersOffset;
	
	 property mixin(DataList!("firstSublist", 
MyBinarySubStructAForA, firstSublistMembersCount, 
firstSublistMembersOffset));
	 property mixin(DataList!("secondSublist", 
MyBinarySubStructBForA, secondSublistMembersCount, 
secondSublistMembersOffset));
}

...

MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr;
Sep 05
next sibling parent John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:
 Hi, everyone. I'm pretty new to D and trying my first steps in 
 it. Currently I'm trying to port some code from C/C++ with 
 pretty weird data structures and don't like idea of making 
 boilerplate functions for accessing sub-lists in main binary 
 structure (lets not talk about it's design, it's from legacy 
 thing I want to deal with and I can't change the format itself 
 or writing a lot of additional parsing code). With hour I spent 
 on trying I've ended with what you may see below, but I wonder 
 if there is more pleasant variant of re-writing that template 
 as I haven't found anything better in either books or online 
 documentation yet:

 template DataList(const char[] listName, T, alias dataOffset, 
 alias listLength)
 {
 	const char[] DataList = format(q{
 		%s[] %s()
 		{
 			return (cast(%s *)(cast(byte *)(&this) + %s))[0 .. %s];
 		}
 	}, T.stringof, listName, T.stringof, dataOffset.stringof, 
 listLength.stringof);
 }

 struct MyBinarySubStructAForA
 {
 	int someIntegerFieldA;
 	float someFloatFieldA;
 }

 struct MyBinarySubStructBForA
 {
 	int someIntegerFieldB;
 	float someFloatFieldB;
 }

 struct MyBinaryStructA
 {
 	int firstSublistMembersCount;
 	int firstSublistMembersOffset;

 	int secondSublistMembersCount;
 	int secondSublistMembersOffset;
 	
 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));
 	 property mixin(DataList!("secondSublist", 
 MyBinarySubStructBForA, secondSublistMembersCount, 
 secondSublistMembersOffset));
 }

 ...

 MyBinaryStructA *binaryData = cast(MyBinaryStructA 
 *)fileData.ptr;
mixin template DataList(string listName, T, alias dataOffset, alias listLength) { mixin( q{T[] } ~ listName ~ q{() property { return (cast(T*)(cast(ubyte*)&this + dataOffset))[0 .. listLength]; }} ); } struct MyBinarySubStructAForA { int someIntegerFieldA; float someFloatFieldA; } struct MyBinarySubStructBForA { int someIntegerFieldB; float someFloatFieldB; } struct MyBinaryStructA { int firstSublistMembersCount; int firstSublistMembersOffset; int secondSublistMembersCount; int secondSublistMembersOffset; mixin DataList!("firstSublist", MyBinarySubStructAForA, firstSublistMembersCount, firstSublistMembersOffset); mixin DataList!("secondSublist", MyBinarySubStructBForA, secondSublistMembersCount, secondSublistMembersOffset); }
Sep 05
prev sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:
 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));
I don't think string mixins are required here. It seems just template is more simple. T[] getBytesAs(T, alias length, alias offset)() { return (cast(T *)(cast(byte *)(&this) + offset))[0 .. length]; } struct MyBinaryStructA { ... alias firstList = getBytesAs!(MyBinarySubStructAForA, firstSublistMembersCount, firstSublistMembersOffset); alias secondList = getBytesAs!(MyBinarySubStructBForA, secondSublistMembersCount, secondSublistMembersOffset); } unittest { ... MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr; auto a = binaryData.firstList; }
Sep 05
next sibling parent reply Computermatronic <computermatronic gmail.com> writes:
On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:
 On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:
 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));
I don't think string mixins are required here. It seems just template is more simple. T[] getBytesAs(T, alias length, alias offset)() { return (cast(T *)(cast(byte *)(&this) + offset))[0 .. length]; } struct MyBinaryStructA { ... alias firstList = getBytesAs!(MyBinarySubStructAForA, firstSublistMembersCount, firstSublistMembersOffset); alias secondList = getBytesAs!(MyBinarySubStructBForA, secondSublistMembersCount, secondSublistMembersOffset); } unittest { ... MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr; auto a = binaryData.firstList; }
I find it very strange that this works, as a non-mixin template should not be able to capture the context of where it was instantiated. If you take the alias template parameters out it behaves how it should (that is an error message saying this is not accessible).
Sep 05
next sibling parent crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
wrote:
 I find it very strange that this works, ...
I'm too. :) In any case, it's not a big deal. It's possible to modify this template to transfer `this` as the first parameter and modify its usage accordingly.
Sep 05
prev sibling next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
wrote:
 On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:
 [...]
I find it very strange that this works, as a non-mixin template should not be able to capture the context of where it was instantiated. If you take the alias template parameters out it behaves how it should (that is an error message saying this is not accessible).
https://issues.dlang.org/show_bug.cgi?id=17809
Sep 05
parent crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 14:43:19 UTC, John Colvin wrote:
 On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
 wrote:
 On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:
 [...]
I find it very strange that this works, as a non-mixin template should not be able to capture the context of where it was instantiated. If you take the alias template parameters out it behaves how it should (that is an error message saying this is not accessible).
https://issues.dlang.org/show_bug.cgi?id=17809
What kind of harm can this be? Can this be considered a feature?
Sep 05
prev sibling parent reply Corey Lubin <i-fear-the-spam coreylub.in> writes:
On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
wrote:
 I find it very strange that this works, as a non-mixin template 
 should not be able to capture the context of where it was 
 instantiated. If you take the alias template parameters out it 
 behaves how it should (that is an error message saying this is 
 not accessible).
Computermatronic (and crimaniak and John Colvin): The behavior is as should be expected and is part of the definition of how templates' alias parameters are supposed to work. The error that occurs when you stop using an alias parameter occurs because you're no longer forcing the capture of an enclosing scope by passing in an alias parameter that requires such a scope; not because the error should have occurred the first time as well. Circumstances where some "context" is captured include: - template instantiations where an alias parameter requires a context in order to resolve the parameter correctly [[the instantiated function/struct/class will effectively become nested within whichever scope is determined to be needed in order to resolve the alias parameter -- whether the alias parameter is needed, used, or not]] - template instantiation of functions/structs/classes whose enclosing templates were declared within a function/struct/class [[the template instance's functions, structs, and classes will effectively become nested within the function/struct/class that the template was declared within -- whether the instantiated function/struct/class's original declaration strictly implied a need for an outer function/struct/class or not]] - and --hrmm... it's 6:20 AM; I haven't yet slept for the night; and I'm supposed to continue preparing to save myself from the strongest hurricane ever seen in the Atlantic Ocean in a couple hours, so I think that's where I'll end this partial list. Some assorted details related to some miscellaneous capture situations: - https://dlang.org/spec/template.html#nested-templates - https://dlang.org/spec/template.html#nested_template_limitation - https://dlang.org/spec/template.html#aliasparameters - https://dlang.org/spec/template.html#limitations - https://dlang.org/spec/function.html#closures
Sep 08
parent Corey Lubin <i-fear-the-spam coreylub.in> writes:
On Friday, 8 September 2017 at 10:27:57 UTC, Corey Lubin wrote:
 On Tuesday, 5 September 2017 at 12:41:45 UTC, Computermatronic 
 wrote:
 I find it very strange that this works, as a non-mixin 
 template should not be able to capture the context of where it 
 was instantiated. If you take the alias template parameters 
 out it behaves how it should (that is an error message saying 
 this is not accessible).
Computermatronic (and crimaniak and John Colvin): The behavior is as should be expected and is part of the definition of how templates' alias parameters are supposed to work. The error that occurs when you stop using an alias parameter occurs because you're no longer forcing the capture of an enclosing scope by passing in an alias parameter that requires such a scope; not because the error should have occurred the first time as well.
I just actually payed attention to what the example code does just now. Looking at the specific example, I suppose I can see what the "strangeness" that's being referred to is. I suppose that even though the context is indeed captured in whole: 1) the fact that the template's original declaration presents like an ordinary freestanding function makes it look weird (like a mixin template function might look weird referring to a context it's not in yet, except more weird because the hinting mixin keyword indicating it could be mixed in anywhere isn't there) 2) the expectation is that the sneakily stolen context will/can only be used for and via the alias parameters it was created captured for, and that the allowing of explicit `this` usage within the definition is too much of a compiler implementation leak? With regards to #2, it is true, of course. (though it makes it easier for implementations) Gotta run.
Sep 08
prev sibling parent reply Void-995 <void995 gmail.com> writes:
On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:
 On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:
 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));
I don't think string mixins are required here. It seems just template is more simple. T[] getBytesAs(T, alias length, alias offset)() { return (cast(T *)(cast(byte *)(&this) + offset))[0 .. length]; } struct MyBinaryStructA { ... alias firstList = getBytesAs!(MyBinarySubStructAForA, firstSublistMembersCount, firstSublistMembersOffset); alias secondList = getBytesAs!(MyBinarySubStructBForA, secondSublistMembersCount, secondSublistMembersOffset); } unittest { ... MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr; auto a = binaryData.firstList; }
Thanks, that definitely working and doesn't require mixin with strings. But while waiting for response I've tried another thing, and I doubt I would able do to that without string now: template MyBinaryStructGenericDataList(string listName, alias Type) { import std.string; const char[] MyBinaryStructGenericDataList = format(q{ int %sCount; int %sOffset; property %s[] %s() const { return (cast(%s *)(cast(ubyte *)(&this) + %sOffset))[0 .. %sCount]; } }, listName, listName, Type.stringof, listName, Type.stringof, listName, listName); } struct MyBinarySubStructAForA { int someIntegerFieldA; float someFloatFieldA; } struct MyBinarySubStructBForA { int someIntegerFieldB; float someFloatFieldB; } struct MyBinaryStructA { mixin(MyBinaryStructGenericDataList!("firstSublist", MyBinarySubStructAForA)); mixin(MyBinaryStructGenericDataList!("secondSublist", MyBinarySubStructBForA)); } ... MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr; ...
Sep 05
next sibling parent Computermatronic <computermatronic gmail.com> writes:
On Tuesday, 5 September 2017 at 12:54:20 UTC, Void-995 wrote:
 On Tuesday, 5 September 2017 at 12:20:14 UTC, crimaniak wrote:
 On Tuesday, 5 September 2017 at 11:08:57 UTC, Void-995 wrote:
 	 property mixin(DataList!("firstSublist", 
 MyBinarySubStructAForA, firstSublistMembersCount, 
 firstSublistMembersOffset));
I don't think string mixins are required here. It seems just template is more simple. T[] getBytesAs(T, alias length, alias offset)() { return (cast(T *)(cast(byte *)(&this) + offset))[0 .. length]; } struct MyBinaryStructA { ... alias firstList = getBytesAs!(MyBinarySubStructAForA, firstSublistMembersCount, firstSublistMembersOffset); alias secondList = getBytesAs!(MyBinarySubStructBForA, secondSublistMembersCount, secondSublistMembersOffset); } unittest { ... MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr; auto a = binaryData.firstList; }
Thanks, that definitely working and doesn't require mixin with strings. But while waiting for response I've tried another thing, and I doubt I would able do to that without string now: template MyBinaryStructGenericDataList(string listName, alias Type) { import std.string; const char[] MyBinaryStructGenericDataList = format(q{ int %sCount; int %sOffset; property %s[] %s() const { return (cast(%s *)(cast(ubyte *)(&this) + %sOffset))[0 .. %sCount]; } }, listName, listName, Type.stringof, listName, Type.stringof, listName, listName); } struct MyBinarySubStructAForA { int someIntegerFieldA; float someFloatFieldA; } struct MyBinarySubStructBForA { int someIntegerFieldB; float someFloatFieldB; } struct MyBinaryStructA { mixin(MyBinaryStructGenericDataList!("firstSublist", MyBinarySubStructAForA)); mixin(MyBinaryStructGenericDataList!("secondSublist", MyBinarySubStructBForA)); } ... MyBinaryStructA *binaryData = cast(MyBinaryStructA *)fileData.ptr; ...
Yes, there is definately no other way but to use string mixins here, however if you switch to a mixin template, and put the string mixin inside of that, you don't have to do text replacement to use the template parameters. Example: mixin template GenericDataList(string name, T) { mixin(` int `~name~`SublistCount; int `~name~`SublistOffset; property auto `~name~`Sublist() { return (cast(T*)(cast(byte*)&this + `~name~`SublistOffset))[0..`~name~`SublistCount]; }`); } struct MyBinarySubStructAForA { int someIntegerFieldA; float someFloatFieldA; } struct MyBinarySubStructBForA { int someIntegerFieldB; float someFloatFieldB; } struct MyBinaryStructA { mixin MyBinaryStructGenericDataList!("first", MyBinarySubStructAForA); mixin MyBinaryStructGenericDataList!("second", MyBinarySubStructBForA); }
Sep 05
prev sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 12:54:20 UTC, Void-995 wrote:
 Thanks, that definitely working and doesn't require mixin with 
 strings. But while waiting for response I've tried another 
 thing, and I doubt I would able do to that without string now:
...
 		int %sCount;
 		int %sOffset;
Yes, you can make such custom names only with string mixin, as I know. Computermatronic gives a good example how to do it simple way. I can give two additional advises: as I understand offsets as sizes are fixed so think about using unions to access sub structures instead of these auto-generated methods; I have a long history of using tricks like this starting from CHAIN MERGE usage in GW-BASIC to add some functions and for now such things are definitely code smell for me. There is always a way to do the same, but better.
Sep 05
parent reply Void-995 <void995 gmail.com> writes:
On Tuesday, 5 September 2017 at 14:01:02 UTC, crimaniak wrote:
 On Tuesday, 5 September 2017 at 12:54:20 UTC, Void-995 wrote:
 Thanks, that definitely working and doesn't require mixin with 
 strings. But while waiting for response I've tried another 
 thing, and I doubt I would able do to that without string now:
...
 		int %sCount;
 		int %sOffset;
Yes, you can make such custom names only with string mixin, as I know. Computermatronic gives a good example how to do it simple way. I can give two additional advises: as I understand offsets as sizes are fixed so think about using unions to access sub structures instead of these auto-generated methods; I have a long history of using tricks like this starting from CHAIN MERGE usage in GW-BASIC to add some functions and for now such things are definitely code smell for me. There is always a way to do the same, but better.
Using unions? <Type>Count and <Type>Offset are different depending on input data, so the address where they are is varying depending on which file I've loaded. Or I didn't get what you meant.
Sep 05
parent reply crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 14:55:21 UTC, Void-995 wrote:

 Using unions? <Type>Count and <Type>Offset are different 
 depending on input data, so the address where they are is 
 varying depending on which file I've loaded. Or I didn't get 
 what you meant.
Yes, so you need separate union type for every type of input data. But these values are known in compile time, so you can do mixin, creating the union for each type like this: union MyBinaryStructA { ubyte[...] asBytes; struct asStruct { align(1): ubyte[...] dummy1; // padding before firstList MyBinarySubStructAForA[...] firstList; ubyte[...] dummy2; // padding between lists MyBinarySubStructBForA[...] secondList; ... } } unittest { MyBinaryStructA a; a.asBytes = someBinarySourceForA; // fill from unstructured source auto item = a.asStruct.firstList[1]; // access to structure element }
Sep 05
parent reply Void-995 <void995 gmail.com> writes:
On Tuesday, 5 September 2017 at 21:18:11 UTC, crimaniak wrote:
 On Tuesday, 5 September 2017 at 14:55:21 UTC, Void-995 wrote:

 Using unions? <Type>Count and <Type>Offset are different 
 depending on input data, so the address where they are is 
 varying depending on which file I've loaded. Or I didn't get 
 what you meant.
Yes, so you need separate union type for every type of input data. But these values are known in compile time, so you can do mixin, creating the union for each type like this: union MyBinaryStructA { ubyte[...] asBytes; struct asStruct { align(1): ubyte[...] dummy1; // padding before firstList MyBinarySubStructAForA[...] firstList; ubyte[...] dummy2; // padding between lists MyBinarySubStructBForA[...] secondList; ... } } unittest { MyBinaryStructA a; a.asBytes = someBinarySourceForA; // fill from unstructured source auto item = a.asStruct.firstList[1]; // access to structure element }
The whole thing is that I don't know either padding nor elements count. Those values are read from file. First you need to read whole file data and cast those bytes as pointer to file header structure. Offset of each list is bigger then size of the parent structure. Length of each list vary from 0 to whatever.
Sep 05
parent crimaniak <crimaniak gmail.com> writes:
On Tuesday, 5 September 2017 at 22:51:45 UTC, Void-995 wrote:
 The whole thing is that I don't know either padding nor 
 elements count. Those values are read from file. ...
Sorry, I was not attentive enough. Yes, you can't use unions in this case.
Sep 06