www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Whole source-tree statefull preprocessing, notion of a whole program

reply Boris-Barboris <ismailsiege gmail.com> writes:
Hello! It's a bit long one, I guess, but I'd like to have some 
discussion of topic. I'll start with a concrete use case:

For the sake of entertainment, I tried to wrote generic 
configuration management class. I was inspired by oslo_config 
python package that I have to deal with on work. I started with:

# module config

class Config
{
     this(string filename) { ... }
     void save() abstract;
     void load() abstract;
     // Some implementation, json for example
}

abstract class ConfigGroup(string group_name_par)
{
     static immutable group_name = group_name_par;
     protected Config CONF;
     this(Config config) { CONF = config; }

     mixin template ConfigField(T, string opt_name)
	{
		mixin(" property " ~ T.stringof ~ " " ~ opt_name ~
			  "() { return CONF.root[\"" ~ group_name ~ "\"][\"" ~ 
opt_name ~
			  "\"]." ~ json_type(T.stringof) ~ "; }");
		mixin(" property " ~ T.stringof ~ " " ~ opt_name ~
			  "(" ~ T.stringof ~ " value) { return CONF.root[\"" ~ 
group_name
               ~ "\"][\"" ~ opt_name ~ "\"]." ~ 
json_type(T.stringof) ~ "(value); }");
	}
}

# module testconfig

private class TestConfigGroup: ConfigGroup!("testGroup")
{
	this(Config config) { super(config); }

	mixin ConfigField!(string, "some_string_option");
	mixin ConfigField!(double, "some_double_option");
}


... aand I stopped. And here are the blockers I saw:

1). I had to save template parameter group_name_par into 
group_name. Looks like template mixin doesn't support closures. A 
minor inconvenience, I would say, and it's not what I would like 
to talk about.
2). After preprocessing I wish to have fully-typed, safe and fast 
Config class, that contains all the groups I defined for it in 
it's body, and not as references. I don't want pointer lookup 
during runtime to get some field. This is actually quite a 
problem for D:
     2.1). Looks like mixin is the only instrument to extend class 
body. Obvious solution would be a loop that mixins some 
definitions from some compile-time known array, meybe even string 
array. And the pretties of all ways - so that such array will 
contain module names and class names of all ConfigGroup 
derivatives defined in whole program (absolute madman). Said 
array could be appended in compile-time by every derivative of 
ConfigGroup.
     2.2) Sweet dream of 2.1 is met with absence of tools to 
create and manipulate state during preprocessing. For example:

     immutable string[] primordial = [];  // maybe some special 
qualifier instead
                                          // of immutable will be 
better. Even
                                          // better if it shifts 
to immutable
                                          // during run-time
     premixin template (string toAdd) { primordial ~= toAdd; } // 
for example
     mixin template Populate
     {
         foreach (s; primordial)
             mixin("int " ~ s);    // create some int field
     }
     class Populated { mixin Populate; }
     // another module
     premixin("field1")  // evaluated in preprocessor in
     premixin("field2")  // order of definition
     Populated p = new Populated;
     p.field1 = 3;

         By "premixin" I mean that all such operations are 
performed in our special preprocessor stage, that is completed 
before mixins we already have now start to do their jobs.

     2.3) There is strong C ancestry in D. The one regarding 
compilation being performed on translation units (.d source 
files) is, in my opinion, quite devastating. I don't know about 
you guys, but in 2017 I compile programs. I don't care about 
individual object files and linker shenanigans, for me it's the 
whole program that matters, and object files are just the way C 
does it's thing. You definetly must respect it while interfacing 
with it, but that's about it. Correct me if I'm wrong, but 
departure from C's compile process (CLI is not the cause here I 
believe) allowed C# to encorporate "partial" classes, wich are a 
wonderfull concept - class can be extended only volunteeringly 
(like in D, where we need to willingly write mixin to change 
definition), and localized source code changes: when project 
functionality is extended, old code base can sometimes remain 
completely untouched (this is huge for very big projects IMO). I 
will not deny, however, that readability of such code suffers. As 
a counter-argument, relationships between portion of the class 
and other code are usually local, in a way that this class part's 
fields are used by source code in this folder and basically 
nowhere else.
         But what's done is done, I understand. However, I 
believe, preprocessor still has hope for it, and can be 
generalized to whole source tree without throwing old toolchain 
out of the window. In the way that would allow "primordial" 
string array from the example above to be the same for all 
translation units after preprocessing is done.
     2.4) Original configuration management example would also 
require the ability to import definitions cyclically. Module A 
containing ConfigGroupConcrete instantiation imports module B 
where Config is defined, wich will require B to import A in order 
to access ConfigGroupConcrete definition. Yet another stone in 
C's garden, yes. You could, for example, pass whole 
ConfigGroupConcrete body as a string and mixin it there, but then 
you would require to automatically build such string, and at this 
point you're better off with some kind of templating language. 
And templating languages make thing even less readable IMO, while 
simply being a crutches to replace language preprocessors, that 
don't follow industry needs. I do believe such case is out of 
reach until preprocessing is done on whole program united.


To conclude, I'll summarize my questions:
1). Is there a compiled language that is capable of the 
abovementiond tricks, without resorting to external templating 
meta-languages?
2). How deep the rabbit hole goes in terms of complexity of 
preprocessor modifications required? And for DMD in general?
3). How are cyclic module imports handled currently in D?
4). Is there hope that it's possible to do in, say, a year? I 
don't mind trying to implement it myself, but I don't want to 
invest time in thing that is so conceptually out of plane that 
will simply be too destructive for current compiler environment.
Apr 08
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 8 April 2017 at 10:11:11 UTC, Boris-Barboris wrote:
 1). I had to save template parameter group_name_par into 
 group_name. Looks like template mixin doesn't support closures. 
 A minor inconvenience, I would say, and it's not what I would 
 like to talk about.
Template mixins' scope is the expansion scope, not the declaration scope. (This is useful in some situations, but recently I have been mostly avoiding template mixins.)
 2). After preprocessing I wish to have fully-typed, safe and 
 fast Config class, that contains all the groups I defined for 
 it in it's body, and not as references. I don't want pointer 
 lookup during runtime to get some field.
Looks like your current implementation does not go in that direction, seeing as it uses properties for field access. For such tasks, I would suggest to split the representations (native data, DOM tree, JSON strings etc.) from the transformations (serialization and parsing/emitting JSON). E.g. your example could be represented as: struct Config { struct TestGroup { string some_string_option; double some_double_option; } TestGroup testGroup; } Then, separate code for serializing/deserializing this to/from a DOM or directly to/from JSON. Individual components' configuration can be delegated to their components; their modules could contain public struct definitions that you can add to the global Config struct, which describes the configuration of the entire application. I've used this pattern successfully in some projects, incl. Digger: https://github.com/CyberShadow/Digger/blob/master/config.d#L31-L36
     2.2) Sweet dream of 2.1 is met with absence of tools to 
 create and manipulate state during preprocessing. For example:
I understand that you seem to be looking for a way to change types (definitions in general) inside modules you import. This is problematic from several aspects, such as other modules depending on that module may find that the definitions "change under their feet". In D, once a type is declared and its final curly brace is closed, you will know that its definition will remain the same from anywhere in the program. D's answer to partial classes is UFCS, however this does not allow "adding" fields, only methods.
     2.4) Original configuration management example would also 
 require the ability to import definitions cyclically. Module A 
 containing ConfigGroupConcrete instantiation imports module B 
 where Config is defined, wich will require B to import A in 
 order to access ConfigGroupConcrete definition.
I don't really understand what you mean here, but D does allow cyclic module imports. It is only forbidden when more than one module inside any cycle has static constructors, because then it is not possible to determine the correct initialization order.
 To conclude, I'll summarize my questions:
 1). Is there a compiled language that is capable of the 
 abovementiond tricks, without resorting to external templating 
 meta-languages?
I don't know of any. For D, I suggest trying different approaches / paradigms.
 2). How deep the rabbit hole goes in terms of complexity of 
 preprocessor modifications required? And for DMD in general?
So far, I had not heard of any D project that requires preprocessing of D code. I think D's metaprogramming has enough solutions to choose from for the vast majority of conceivable situations where other languages would call for a preprocessor.
 4). Is there hope that it's possible to do in, say, a year? I 
 don't mind trying to implement it myself, but I don't want to 
 invest time in thing that is so conceptually out of plane that 
 will simply be too destructive for current compiler environment.
I suggest that you examine how established D projects deal with similar situations.
Apr 08
parent reply Boris-Barboris <ismailsiege gmail.com> writes:
On Saturday, 8 April 2017 at 13:09:59 UTC, Vladimir Panteleev 
wrote:
 On Saturday, 8 April 2017 at 10:11:11 UTC, Boris-Barboris wrote:
 2). After preprocessing I wish to have fully-typed, safe and 
 fast Config class, that contains all the groups I defined for 
 it in it's body, and not as references. I don't want pointer 
 lookup during runtime to get some field.
Looks like your current implementation does not go in that direction, seeing as it uses properties for field access.
Am i mistaken in assumption that such simple getter property will be optimized to direct field access? Anyways, that's minor detail.
 For such tasks, I would suggest to split the representations 
 (native data, DOM tree, JSON strings etc.) from the 
 transformations (serialization and parsing/emitting JSON). E.g. 
 your example could be represented as:

 struct Config
 {
     struct TestGroup
     {
         string some_string_option;
         double some_double_option;
     }
     TestGroup testGroup;
 }

 Then, separate code for serializing/deserializing this to/from 
 a DOM or directly to/from JSON.

 Individual components' configuration can be delegated to their 
 components; their modules could contain public struct 
 definitions that you can add to the global Config struct, which 
 describes the configuration of the entire application. I've 
 used this pattern successfully in some projects, incl. Digger: 
 https://github.com/CyberShadow/Digger/blob/master/config.d#L31-L36
Ok, that's nice, but it still requires manual inclusion of such field into global config struct. Some "compile-time callback" system still would scale better in my opinion.
     2.2) Sweet dream of 2.1 is met with absence of tools to 
 create and manipulate state during preprocessing. For example:
I understand that you seem to be looking for a way to change types (definitions in general) inside modules you import. This is problematic from several aspects, such as other modules depending on that module may find that the definitions "change under their feet".
As expected since class that allows itself to be modified in compile-time, always does so explicitly via mixin. Most of the times such manipulation is used to extend functionality (add field, plugin, method) without removing or modifying existing ones. And if the names conflict, we get nice compile-time error anyways.
 In D, once a type is declared and its final curly brace is 
 closed, you will know that its definition will remain the same 
 from anywhere in the program.
That's kinda my point - definition needs to stay the same because it's built by compiler as many times as there are transtaltion units, because evil old C grandpa.
 D's answer to partial classes is UFCS, however this does not 
 allow "adding" fields, only methods.
Adding fields, or, generally, objects \ collections of objects, is the main use case. Adding methods in my experience is rare scenario.
     2.4) Original configuration management example would also 
 require the ability to import definitions cyclically. Module A 
 containing ConfigGroupConcrete instantiation imports module B 
 where Config is defined, wich will require B to import A in 
 order to access ConfigGroupConcrete definition.
I don't really understand what you mean here, but D does allow cyclic module imports. It is only forbidden when more than one module inside any cycle has static constructors, because then it is not possible to determine the correct initialization order.
Exactly what I wanted to know, thank you.
 To conclude, I'll summarize my questions:
 1). Is there a compiled language that is capable of the 
 abovementiond tricks, without resorting to external templating 
 meta-languages?
I don't know of any. For D, I suggest trying different approaches / paradigms.
 2). How deep the rabbit hole goes in terms of complexity of 
 preprocessor modifications required? And for DMD in general?
So far, I had not heard of any D project that requires preprocessing of D code. I think D's metaprogramming has enough solutions to choose from for the vast majority of conceivable situations where other languages would call for a preprocessor.
 4). Is there hope that it's possible to do in, say, a year? I 
 don't mind trying to implement it myself, but I don't want to 
 invest time in thing that is so conceptually out of plane that 
 will simply be too destructive for current compiler 
 environment.
I suggest that you examine how established D projects deal with similar situations.
Thank you.
Apr 08
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Saturday, 8 April 2017 at 14:20:49 UTC, Boris-Barboris wrote:
 Looks like your current implementation does not go in that 
 direction, seeing as it uses properties for field access.
Am i mistaken in assumption that such simple getter property will be optimized to direct field access? Anyways, that's minor detail.
I don't know the type of CONF.root, but from the usage syntax in your example, it looks like an associative array. Associative array lookup will be slower than simply accessing a variable.
 Individual components' configuration can be delegated to their 
 components; their modules could contain public struct 
 definitions that you can add to the global Config struct, 
 which describes the configuration of the entire application. 
 I've used this pattern successfully in some projects, incl. 
 Digger: 
 https://github.com/CyberShadow/Digger/blob/master/config.d#L31-L36
Ok, that's nice, but it still requires manual inclusion of such field into global config struct.
Yes; in my opinion, I think that's desirable because it is aligned with the unidirectional flow of information from higher-level components to lower-level ones, and does not impose a particular configuration framework onto the lower-level components (they only need to declare their configuration in terms of a POD type).
 Some "compile-time callback" system still would scale better in 
 my opinion.
A similar effect can be achieved by allowing components to register themselves in a static constructor (not at compile-time, but at program start-up).
 I understand that you seem to be looking for a way to change 
 types (definitions in general) inside modules you import. This 
 is problematic from several aspects, such as other modules 
 depending on that module may find that the definitions "change 
 under their feet".
As expected since class that allows itself to be modified in compile-time, always does so explicitly via mixin. Most of the times such manipulation is used to extend functionality (add field, plugin, method) without removing or modifying existing ones. And if the names conflict, we get nice compile-time error anyways.
Then you have problems such as the instance size of a class changing depending on whether the code that requires the instance size is seen by the compiler before the code that modifies the instance size. I think it would cause complicated design problems that limit the scalability of the language. Even without such features, DMD had to go through a number of bugs to iron out the correct semantics of evaluating types (e.g. with "typeof(this).sizeof" inside a struct declaration, or recursive struct template instantiations).
 In D, once a type is declared and its final curly brace is 
 closed, you will know that its definition will remain the same 
 from anywhere in the program.
That's kinda my point - definition needs to stay the same because it's built by compiler as many times as there are transtaltion units, because evil old C grandpa.
I think this is not about technical limitations, but intentional design choices. Allowing types to be modified post-declaration invalidates many contracts and assumptions that code may have, and make it harder to reason about the program as a whole. Compare with e.g. INTERCAL's COMEFROM instruction.
 D's answer to partial classes is UFCS, however this does not 
 allow "adding" fields, only methods.
Adding fields, or, generally, objects \ collections of objects, is the main use case. Adding methods in my experience is rare scenario.
UFCS is widely used in D for component programming: http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321
Apr 08
parent Boris-Barboris <ismailsiege gmail.com> writes:
On Saturday, 8 April 2017 at 17:57:11 UTC, Vladimir Panteleev 
wrote:

 Yes; in my opinion, I think that's desirable because it is 
 aligned with the unidirectional flow of information from 
 higher-level components to lower-level ones, and does not 
 impose a particular configuration framework onto the 
 lower-level components (they only need to declare their 
 configuration in terms of a POD type).
...
 A similar effect can be achieved by allowing components to 
 register themselves in a static constructor (not at 
 compile-time, but at program start-up).
That is definetly possible and, I would say, trivial, and this is the most popular way. However, any run-time registration implies run-time collections to iterate over, with obvious performance drawbacks (minor ones in this case). We are not using the information we have a-priory, in compile time, and make CPU pay for it instead (either because we are too lazy (too busy) to update sources of higher-level components (while making them a mess), or just because our language lacks expressibility, wich is my point). I side with another set of virtues. Source code consists of files. Files contain related data, concepts, functionality, whatever. Relations between those entities must by no means be unidirectional. What direction can you impose to concept "For every configurable entity, be that package, module, or class, I need to have fields in global configuration singleton"? IMO program has good architecture, when during extensive development for arbitrary group of programmers it takes little time and effort to make usefull changes. It is achieved by extensive problem field research, use of abstraction to fight complexities, yada yada... Part of it is to make sure, that you can extend functionality easily. Adding new subclass in one file and registering it in two others is not hard. But there is no fundamental reason for it to not be easier: just add subclass and slap some fancy attribute on it, or add some preprocessor-related field or function in it's body. Onde-directional flow is a consequence, not a principle. It's because languages were made this way we are used to it. When high-level concept or idea willingly implies feedback from it's users, there is little reason to forbid it. Especially when it actually improves development iteration times, lowers risks of merge conflicts etc. Look at this mess: https://github.com/Boris-Barboris/AtmosphereAutopilot/blob/master/AtmosphereAutopilot/GUI/AutoGui.cs#L190 It's caching code for some C# reflection-based GUI I wrote some time ago, that defines attribute to mark class fields with in order to draw them in pretty little debug window. Why do I have to do this? I've got all information right in the source. All classes that will be drawn using this GUI module are there, in text, accessible to build system. Why can't I just write clean code, that doesn't involve double or tripple associative array dispatch on runtime-reflected list of subclasses and attribute-marked fields? Answer is simple - language lacks expressibility. I provide drawing functionality in my module. It is generic, it is virtuous, it is concentrated in one file, it speeds up development. However, it needs to see it's client, beneficient, in order to draw him. And it just doesn't. Because C#, and you need to do runtime reflections. Yet again, by throwing away information you already have and making CPU reconstruct it again during runtime, over and over. Yes, all things I describe can be done efficiently by writing a lot of boilerplate code or using some text-templating magic. I just don't see why languages can't have that functionality built-in.
 Then you have problems such as the instance size of a class 
 changing depending on whether the code that requires the 
 instance size is seen by the compiler before the code that 
 modifies the instance size. I think it would cause complicated 
 design problems that limit the scalability of the language. 
 Even without such features, DMD had to go through a number of 
 bugs to iron out the correct semantics of evaluating types 
 (e.g. with "typeof(this).sizeof" inside a struct declaration, 
 or recursive struct template instantiations).
I agree. I think such mechanisms must be applied very early, hence "premixin".
 I think this is not about technical limitations, but 
 intentional design choices. Allowing types to be modified 
 post-declaration invalidates many contracts and assumptions 
 that code may have, and make it harder to reason about the 
 program as a whole. Compare with e.g. INTERCAL's COMEFROM 
 instruction.
I still don't see the problem. Declaration will contain constructs that indicate that it will be changed, like, for example, "premixin" that iterates over array and adds fields. I have no doubt human can read this allright. Indeed, question of ordering is important. Well, we have goto, it's not like sky dropped down on us. COMEFROM breaks logical time flow. I only want two-staged preprocessing, when in first stage I can create and manipulate some simple, visible to preprocessor state, even if it consists of only immutable base types and arrays of those, and then being able to use that state as immutable variables in next stages we already have. We already can populate class with fields from immutable string array. All I'm wanting is ability to populate this array using preprocessor directives from across whole compiled program, before all other complex stuff starts. I think that would be beautiful.
 UFCS is widely used in D for component programming:

 http://www.drdobbs.com/architecture-and-design/component-programming-in-d/240008321
I'm not stating the opposite, just sharing what I encountered on work or during programming for fun - I mostly needed to add fields. People may feel otherwise, but I don't see this concept harming them in any way.
Apr 08