www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP81: Writing files at compile time

reply "JDemler" <jakob.demler stud-mail.uni-wuerzburg.de> writes:
Triggered by the original forum thread I wrote a DIP to further 
explain the idea of the writing files at compile time feature and 
its implications.

http://wiki.dlang.org/DIP81

Please discuss!
Aug 12 2015
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to further 
 explain the idea of the writing files at compile time feature 
 and its implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
1. How do you order imports and exports? 2. How do you support parallel compilation on a cluster?
Aug 12 2015
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
I've been looking at a company's build system recently and it 
makes me think this is a bad idea: we have enough problems 
tracking import dependencies and changes as it is without other 
files being written in the middle of the process too.

I'd prefer it if these were always done as separate steps so it 
can be more easily integrated into other build scripts. (Build 
systems and scripts suck btw, but they suck less than the 
confusion caused when files can be read and written arbitrarily 
by the compiler, which already has hidden dependencies).
Aug 12 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:
 I've been looking at a company's build system recently and it 
 makes me think this is a bad idea: we have enough problems 
 tracking import dependencies and changes as it is without other 
 files being written in the middle of the process too.
Yes, you cannot use the file system for this. You could make it work by having a central write-once key-value database, then block all imports from non-existing keys until they become available. And error out if either keys never materialize or if a key is written twice.
Aug 12 2015
parent reply "JDemler" <jakob.demler stud-mail.uni-wuerzburg.de> writes:
On Wednesday, 12 August 2015 at 19:12:53 UTC, Ola Fosheim Grøstad 
wrote:
 On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe 
 wrote:
 I've been looking at a company's build system recently and it 
 makes me think this is a bad idea: we have enough problems 
 tracking import dependencies and changes as it is without 
 other files being written in the middle of the process too.
Yes, you cannot use the file system for this. You could make it work by having a central write-once key-value database, then block all imports from non-existing keys until they become available. And error out if either keys never materialize or if a key is written twice.
I see your point. But not using the file system would imply that the debugging and transparency advantages would not work. Maybe a combination of both could work? A central database that tracks which files have been generated and which have not, but the imported code still lies on the file system. Although that seems overly complex and would imply a differantiation in the import syntax between generated and non generated files.
Aug 12 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 12 August 2015 at 20:21:06 UTC, JDemler wrote:
 Maybe a combination of both could work? A central database that 
 tracks which files have been generated and which have not, but 
 the imported code still lies on the file system.
 Although that seems overly complex and would imply a 
 differantiation in the import syntax between generated and non 
 generated files.
I think you should forget that there is a filesystem. You may have at least 4 storage areas: 1. a source bundle 2. generated source bundle 3. an output bundle 4. perhaps a temporary storage area So you need to differentiate between those. You don't have to differentiate between generated and non-generated if you only allow keys in 2 that do not exist in 1. When you reference a module you simply look at 1. first, if it does not exist you try 2. If you can write files to the output bundle (like .ini or .xml files etc) you might also need specify the mime-type.
Aug 12 2015
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
And for this to work you also need a highly concurrent compiler.
Aug 12 2015
prev sibling parent reply "JDemler" <jakob.demler stud-mail.uni-wuerzburg.de> writes:
On Wednesday, 12 August 2015 at 20:39:31 UTC, Ola Fosheim Grøstad 
wrote:
 On Wednesday, 12 August 2015 at 20:21:06 UTC, JDemler wrote:
 Maybe a combination of both could work? A central database 
 that tracks which files have been generated and which have 
 not, but the imported code still lies on the file system.
 Although that seems overly complex and would imply a 
 differantiation in the import syntax between generated and non 
 generated files.
I think you should forget that there is a filesystem. You may have at least 4 storage areas: 1. a source bundle 2. generated source bundle 3. an output bundle 4. perhaps a temporary storage area So you need to differentiate between those. You don't have to differentiate between generated and non-generated if you only allow keys in 2 that do not exist in 1. When you reference a module you simply look at 1. first, if it does not exist you try 2. If you can write files to the output bundle (like .ini or .xml files etc) you might also need specify the mime-type.
I am not sure if I understand your idea correctly: The compiler would in case of an export(name, content) write the content both to a internal database and the filesystem and then only use the internal one? Or could the content of the internal database be copied to the file system at the end of the compilation process? Or is only the output bundle written to the file system?
Aug 12 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 13 August 2015 at 00:54:37 UTC, JDemler wrote:
 I am not sure if I understand your idea correctly:
 The compiler would in case of an export(name, content) write 
 the content both to a internal database and the filesystem and 
 then only use the internal one?
 Or could the content of the internal database be copied to the 
 file system at the end of the compilation process? Or is only 
 the output bundle written to the file system?
The filesystem is a compiler issue and not a language issue, so sure, the compiler could do whatever it wants, flush everything to disk or into a SQL database or… Keep in mind that someone might want to compile on a diskless computer from an in-memory zip-file or similar. I think the important file generation that are relevant to the language would be for the output bundle, so that you e.g. can have a config file in D and generate platform specific files (.ini, .xml etc).
Aug 13 2015
parent reply "JDemler" <jakob.demler stud-mail.uni-wuerzburg.de> writes:
On Thursday, 13 August 2015 at 08:19:15 UTC, Ola Fosheim Grøstad 
wrote:
 On Thursday, 13 August 2015 at 00:54:37 UTC, JDemler wrote:
 I am not sure if I understand your idea correctly:
 The compiler would in case of an export(name, content) write 
 the content both to a internal database and the filesystem and 
 then only use the internal one?
 Or could the content of the internal database be copied to the 
 file system at the end of the compilation process? Or is only 
 the output bundle written to the file system?
The filesystem is a compiler issue and not a language issue, so sure, the compiler could do whatever it wants, flush everything to disk or into a SQL database or… Keep in mind that someone might want to compile on a diskless computer from an in-memory zip-file or similar. I think the important file generation that are relevant to the language would be for the output bundle, so that you e.g. can have a config file in D and generate platform specific files (.ini, .xml etc).
To summarize: We would have a central registry handling all the generated and non generated files. If an import is triggered, this registry is then searched for the imported file. If this file does not already exist, the import blocks. Writing these generated files to the file system would be optional, triggered by a compiler switch and happen after the compilation process is complete. This solves the concurrency problem. But what about non concurrent compiler runs? A block there would never be resolved. I do not know enough about compilers to judge this problem. Can a sane way of compiling the files be found easily? Can the compiler switch to compile something else when it is blocked on an import? And I do not understand why you differentiate between generated source files and generated non source files (.xml, .ini). As both types can be imported (or read at compile time) I do not think we should treat them differently. Also we would need a way to tell the one from the other (file extension? different parameter in the export syntax?). In my understanding storage area 2 and 3 should be merged.
Aug 13 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 13 August 2015 at 09:52:02 UTC, JDemler wrote:
 This solves the concurrency problem. But what about non 
 concurrent compiler runs? A block there would never be 
 resolved. I do not know enough about compilers to judge this 
 problem. Can a sane way of compiling the files be found easily? 
 Can the compiler switch to compile something else when it is 
 blocked on an import?
You need a highly concurrent compiler. If you import in one definition and export in the next definition, then the first one would block and be queued, and the compiler would move on to the next definition. In the actor-model (co-routines/fibers) each definition will spawn a new actor.
 And I do not understand why you differentiate between generated 
 source files and generated non source files (.xml, .ini). As 
 both types can be imported (or read at compile time) I do not 
 think we should treat them differently. Also we would need a 
 way to tell the one from the other (file extension? different 
 parameter in the export syntax?).
 In my understanding storage area 2 and 3 should be merged.
Area 2 is pre-populated with area 1, so they are the same. You need to be able to tell the compiler what files should be included in the output bundle. Actually, I think the file-model is too simple. I think you need a more advanced key-value database so that you can update different fields in the same ".ini file" from different source files.
Aug 13 2015
parent "JDemler" <jakob.demler stud-mail.uni-wuerzburg.de> writes:
On Thursday, 13 August 2015 at 10:06:32 UTC, Ola Fosheim Grøstad 
wrote:
 You need to be able to tell the compiler what files should be 
 included in the output bundle.
While I basically agree that this would be a nice to have feature I do not think it is necessary and it would not make up for the added complexity inside the export syntax. If a specific source file should not be written to the file system one can still use mixins.
 Actually, I think the file-model is too simple. I think you 
 need a more advanced key-value database so that you can update 
 different fields in the same ".ini file" from different source 
 files.
This would a huge amount of complexity to the feature and would not work in general. One can add keys to .ini and maybe .config files but not to a .sql or .css file. Also a append or prepend feature would make it impossible for the compiler to tell when a generated file is finished. Based on this discussion I updated the DIP. Included are now the idea of mixin and template expanding (in the section alternatives) and Ola's solution to the basic problems which I originally missed out on (in the section implementation considerations). All in all I have to admit that the original approach was not designed well enough. It was simple but not functional. Alas, this new approach brings with it a completly different implementation complexity. Nevertheless, the idea of writing files at compile time is still worth discussing about.
Aug 14 2015
prev sibling parent "lobo" <swamplobo gmail.com> writes:
On Wednesday, 12 August 2015 at 18:57:05 UTC, Adam D. Ruppe wrote:
 I've been looking at a company's build system recently and it 
 makes me think this is a bad idea: we have enough problems 
 tracking import dependencies and changes as it is without other 
 files being written in the middle of the process too.

 I'd prefer it if these were always done as separate steps so it 
 can be more easily integrated into other build scripts. (Build 
 systems and scripts suck btw, but they suck less than the 
 confusion caused when files can be read and written arbitrarily 
 by the compiler, which already has hidden dependencies).
+100
Aug 12 2015
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
Please no.
Aug 12 2015
prev sibling next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to further 
 explain the idea of the writing files at compile time feature 
 and its implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
Aug 12 2015
next sibling parent "Nicholas Wilson" <iamthewilsonator hotmail.com> writes:
On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:
 On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 [...]
The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
I think this is a much better idea. Nic
Aug 12 2015
prev sibling next sibling parent reply "JDemler" <jakob.demler stud-mail.uni-wuerzburg.de> writes:
On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:
 On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to 
 further explain the idea of the writing files at compile time 
 feature and its implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
While this might work for very simple and basic mixins, with the combination of TMP and compile time reflection this becomes not only impractical but also impossible. Think again about the vibe.d example: There is one mixin handeling all the template-type combinations. How would such a mixin be expanded? The resulting code differs from template to template and from type to type.
Aug 12 2015
parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 13 August 2015 at 00:58:14 UTC, JDemler wrote:
 On Wednesday, 12 August 2015 at 23:27:16 UTC, Tofu Ninja wrote:
 On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to 
 further explain the idea of the writing files at compile time 
 feature and its implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins. A simple alternative to this would be a flag to the compiler to expand mixins and output the new files. This would also be great for tooling, an IDE could use this and allow you to expand a mixin in place to see what it looks like. And currently all the auto complete engines I have seen for D don't handle mixins very well. Expanding them would make autocompletion a simpler job.
While this might work for very simple and basic mixins, with the combination of TMP and compile time reflection this becomes not only impractical but also impossible. Think again about the vibe.d example: There is one mixin handeling all the template-type combinations. How would such a mixin be expanded? The resulting code differs from template to template and from type to type.
I suppose such an expansion flag would need to expand templates as well, which is still not a bad idea. Templates can be hard to follow sometimes and expanding them out in all their forms could be helpful for debugging and for tooling as well. Though there would be a lot of expansions.
Aug 12 2015
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 13-Aug-2015 02:27, Tofu Ninja wrote:
 On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to further
 explain the idea of the writing files at compile time feature and its
 implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
The benefits of this I see are debugging, actually having the generated files makes it much simpler to see what is going wrong. Otherwise the utility of this can be achieved with string mixins.
pragma(msg, your_mixin);
 A simple alternative to this would be a flag to the compiler to expand
 mixins and output the new files. This would also be great for tooling,
 an IDE could use this and allow you to expand a mixin in place to see
 what it looks like. And currently all the auto complete engines I have
 seen for D don't handle mixins very well. Expanding them would make
 autocompletion a simpler job.
I agree with Ola, something more restricted then filesystem is required. -- Dmitry Olshansky
Aug 12 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-08-13 01:27, Tofu Ninja wrote:

 A simple alternative to this would be a flag to the compiler to expand
 mixins and output the new files. This would also be great for tooling,
 an IDE could use this and allow you to expand a mixin in place to see
 what it looks like. And currently all the auto complete engines I have
 seen for D don't handle mixins very well. Expanding them would make
 autocompletion a simpler job.
I agree, this is much better. I remember the old Eclipse plugin, Descent. It had a compile time view which showed how mixins where expanded and how some language features where lowered, i.e. "scope" to try-catch-finally. -- /Jacob Carlborg
Aug 12 2015
parent "ixid" <adamsibson hotmail.com> writes:
On Thursday, 13 August 2015 at 06:53:14 UTC, Jacob Carlborg wrote:
 On 2015-08-13 01:27, Tofu Ninja wrote:

 A simple alternative to this would be a flag to the compiler 
 to expand
 mixins and output the new files. This would also be great for 
 tooling,
 an IDE could use this and allow you to expand a mixin in place 
 to see
 what it looks like. And currently all the auto complete 
 engines I have
 seen for D don't handle mixins very well. Expanding them would 
 make
 autocompletion a simpler job.
I agree, this is much better. I remember the old Eclipse plugin, Descent. It had a compile time view which showed how mixins where expanded and how some language features where lowered, i.e. "scope" to try-catch-finally.
This would be great but we would need some support for the formatting I think, mixin code tends to be a mess, it would be nice to have some basic auto indent or general FMT style support.
Aug 13 2015
prev sibling next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 13/08/2015 6:37 a.m., JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to further explain
 the idea of the writing files at compile time feature and its implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
Problem: debugging The debugger cannot attach to mixedin code; one has to print the resulting code with pragma(msg, ) to inspect it Fix: Make D interface file generation include the source code e.g. statements. Pros: - Already have most of the code to do this Cons: - None. Problem: compile-speed Even if the resulting code does not change, mixedin code has to be compiled anyway True. Why is this a problem exactly? Problem: scalability If the generated code is needed in two seperated places of the code, it has to be mixedin twice or has to be mixedin into a special module which introduces overhead Not a problem? The result should *theoretically* the return value should be cached. We could even go so far as to have ``mixin(string)`` as the return type. To cache the AST and force compile time only. Perhaps even make it return the assembly code itself for runtime? Problem: transparency As user of a library which relys on compile time code generation one has often no idea what code is generated See debugging problem for the solution. We do not need to add a new language feature based upon these problems and use cases. We can solve it by simply extending one we already have. D interface files. Making them generate the full source code instead of just the interface.
Aug 12 2015
parent reply "JDemler" <jakob.demler stud-mail.uni-wuerzburg.de> writes:
On Thursday, 13 August 2015 at 04:58:06 UTC, Rikki Cattermole 
wrote:
 [...]
So in the vibe.d example we would generate a file diet.di which would include all the generated diet templates of the whole project? Maybe 100 * 50 extra lines of code for a medium sized project? This is not debuggable, understandable or transparent. Especially because the generated code lies inside a file of a library which I should not need to know about. Also I do not understand how we could expand template instatiations. Maybe it would be possible for basic templates that only take types but imagine a template that takes a parse tree as a value. How would an expanded version of that look like?
Aug 13 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 13/08/2015 7:57 p.m., JDemler wrote:
 On Thursday, 13 August 2015 at 04:58:06 UTC, Rikki Cattermole wrote:
 [...]
So in the vibe.d example we would generate a file diet.di which would include all the generated diet templates of the whole project? Maybe 100 * 50 extra lines of code for a medium sized project? This is not debuggable, understandable or transparent. Especially because the generated code lies inside a file of a library which I should not need to know about.
If it is done via D interface files, you would be in control of where it lands. It would not go into a dub package directory that you do not control for your project. You want it to be debuggable, this is. Is it nice and pretty? No. If you are interested in only one code path i.e. specific template arguments to outputted and not the others, then perhaps a pragma can be used to limit it.
 Also I do not understand how we could expand template instatiations.
 Maybe it would be possible for basic templates that only take types but
 imagine a template that takes a parse tree as a value. How would an
 expanded version of that look like?
Template if statement. Also argument overloads. Also remember, the information must exist to expand it. If it didn't, we sure couldn't generate assembly.
Aug 13 2015
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to further 
 explain the idea of the writing files at compile time feature 
 and its implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
I like the idea of being able to write files at compile-time. Is there any progress on this.
Mar 08
parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 08.03.2016 um 11:28 schrieb Stefan Koch:
 On Wednesday, 12 August 2015 at 18:37:40 UTC, JDemler wrote:
 Triggered by the original forum thread I wrote a DIP to further
 explain the idea of the writing files at compile time feature and its
 implications.

 http://wiki.dlang.org/DIP81

 Please discuss!
I like the idea of being able to write files at compile-time. Is there any progress on this.
No progress on the DIP, I made an alternative implementation for vibe.d's Diet templates using multiple `shared static this()` and writing the files at run time: https://github.com/rejectedsoftware/vibe.d/pull/1385 Performance results are very promising. I don't know which way is nicer (or uglier), but at least for this use case, the DIP isn't strictly necessary.
Mar 08