www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A idea to improve diet-ng compile speed!

reply workman <workman gmail.com> writes:
diet-ng is great and I love to use it,  but it is too slow to 
compile.

One idea to speedup is use preprocessor like DMD compile C header 
file with GCC -E.

A diet-ng compiler write in D, translate template file into D 
source code, then mixin from projects.  A runtime diet-ng 
compiler will easaly speed up 1000 times compare to CTFE version, 
and you don't get any performance loss.

A runtime diet-ng compiler also possible compile template into 
dynamic library file,   you edit view files and a file monitor 
will recompile the dynamic library then http server reload it, no 
need recompile full projects.

Add new feature into diet will be more easy, since developer 
don't limit to use CTFE code.

D CTFE is great, but not suit for task like dieg-ng.
Aug 12
next sibling parent reply workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 07:55:08 UTC, workman wrote:
 D CTFE is great, but not suit for task like dieg-ng.
simple code like this, finish in 0.004s for 20 files. ldc2 take hours to finish in product mode. ```c import diet.defs; import diet.dom; import diet.input; import diet.parser; import diet.html; import std.stdio; import std.file; import std.path; int main(string[] args){ foreach(filepath; args[1..$]){ scope filename = baseName(filepath); scope ext = extension(filepath); if( ext != ".dt" ) { writefln("%s extension %s is not .dt", filepath, ext); break; } scope dfile = filepath[0..$-1]; try { scope data = cast(string) read(filepath); scope doc = parseDiet(data, filename); scope code = getHTMLMixin(doc, dietOutputRangeName, HTMLOutputStyle.compact); if( std.file.exists(dfile) && isFile(dfile) ) { scope _code = cast(string) read(dfile); if( _code == code ) continue; } writefln("%s => %s", filepath, dfile); if( is_cached ) { continue; } std.file.write(dfile, code); } catch(Exception e) { writefln("%s throw %s", filepath, e); break; } } return 0; } ``` dieg not support inherit and layout like https://mozilla.github.io/nunjucks/ or https://jinja.palletsprojects.com/en/3.0.x/, to add more advance function CTFE will crashed to compile.
Aug 12
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/21 9:45 AM, workman wrote:
 On Thursday, 12 August 2021 at 07:55:08 UTC, workman wrote:
 D CTFE is great, but not suit for task like dieg-ng.
simple code like this, finish in 0.004s for 20 files. ldc2 take hours to finish in product mode.
From experience, I think the 2 hours is probably not because of diet CTFE interpretation. But I could be wrong, I don't know what your application is like.
 
 dieg not support inherit and layout like 
 https://mozilla.github.io/nunjucks/ or 
 https://jinja.palletsprojects.com/en/3.0.x/, to add more advance 
 function CTFE will  crashed to compile.
diet supports layout inheritance via the `extends` directive. e.g. all my files look like: ```pug extends layout block extraCss // extra css includes block extraJs // extra js includes block content // page content ``` Where `layout.dt` provides the overall layout. -Steve
Aug 12
parent reply workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 13:49:32 UTC, Steven Schveighoffer 
wrote:

 But changing the code means recompiling. And that is not always 
 easy, because diet templates are compiled with specific context 
 and aliases, which are not easy to replicate as a subpart of 
 the project. I have considered splitting my routes out into 
 individual subprojects to see if that helps with the build 
 times.
Mixed d code with views made vibe.d like old PHP code, it hard to maintain if business logic change fast, and not easy to testing. I am consider use a javascript server side render template like On Thursday, 12 August 2021 at 14:04:03 UTC, Steven Schveighoffer wrote:
 On 8/12/21 9:45 AM, workman wrote:
 From experience, I think the 2 hours is probably not because of 
 diet CTFE interpretation. But I could be wrong, I don't know 
 what your application is like.
Not really take hours, but very slow and require a lot memory.
 diet supports layout inheritance via the `extends` directive.

 e.g. all my files look like:

 ```pug
 extends layout
 block extraCss
   // extra css includes
 block extraJs
   // extra js includes
 block content
   // page content
 ```
Thanks for the tips, I will try extends. What will be really nice is some thing like: ```twig {% extends "base.html" %} {% block sidebar %} <h3>Table Of Contents</h3> ... {{ parent() }} {% endblock %} ```
Aug 12
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/21 10:11 AM, workman wrote:
 On Thursday, 12 August 2021 at 13:49:32 UTC, Steven Schveighoffer wrote:
 
 But changing the code means recompiling. And that is not always easy, 
 because diet templates are compiled with specific context and aliases, 
 which are not easy to replicate as a subpart of the project. I have 
 considered splitting my routes out into individual subprojects to see 
 if that helps with the build times.
Mixed d code with views made vibe.d like old PHP code, it hard to maintain if business logic change fast, and not easy to testing. I am consider use a javascript server side render template like On Thursday, 12 August 2021 at 14:04:03 UTC, Steven Schveighoffer wrote:
 On 8/12/21 9:45 AM, workman wrote:
 From experience, I think the 2 hours is probably not because of diet 
 CTFE interpretation. But I could be wrong, I don't know what your 
 application is like.
Not really take hours, but very slow and require a lot memory.
OK. My experience is that precompiling templates and using the cache feature reduced my build time from 38 seconds to 11, and also reduced memory consumption quite a bit. However, you still must build production without these modes. I would like to introduce a production precompilation mode, where the diet processing is done separately, and the code just imports the file without having to checksum it. But I haven't had the need yet. If you have memory constraints that prevent you from building production, this might be a viable path.
 
 
 diet supports layout inheritance via the `extends` directive.

 e.g. all my files look like:

 ```pug
 extends layout
 block extraCss
   // extra css includes
 block extraJs
   // extra js includes
 block content
   // page content
 ```
Thanks for the tips, I will try extends. What will be really nice is some thing like: ```twig {% extends "base.html" %}     {% block sidebar %}         <h3>Table Of Contents</h3>         ...         {{ parent() }}     {% endblock %}     ```
Yes, that is exactly what `extends` does. Same keywords even. See [docs](https://github.com/rejectedsoftware/diet-ng/blob/master/SPEC.md extension-includes) (they are a bit sparse, but there's an example there to help you understand how to do it). -Steve
Aug 12
parent reply workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 14:51:28 UTC, Steven Schveighoffer 
wrote:
 Yes, that is exactly what `extends` does. Same keywords even. 
 See 
 [docs](https://github.com/rejectedsoftware/diet-ng/blob/master/SPEC.md
extension-includes) (they are a bit sparse, but there's an example there to
help you understand how to do it).

 -Steve
Thanks again for the tips. The `extends file(.ext)` is not implement yet, and `include section.dt` is not work like other popular template system ( need super() or parent() ). I am think of use https://mozilla.github.io/nunjucks/ as a client side solution, it is easy to add a server side render solution with quickjs or a nodejs.
Aug 12
parent workman <workman gmail.com> writes:
On Thursday, 12 August 2021 at 15:48:29 UTC, workman wrote:
 On Thursday, 12 August 2021 at 14:51:28 UTC, Steven 
 Schveighoffer wrote:
 Yes, that is exactly what `extends` does. Same keywords even. 
 See 
 [docs](https://github.com/rejectedsoftware/diet-ng/blob/master/SPEC.md
extension-includes) (they are a bit sparse, but there's an example there to
help you understand how to do it).

 -Steve
Thanks again for the tips. The `extends file(.ext)` is not implement yet, and `include section.dt` is not work like other popular template system ( need super() or parent() ). I am think of use https://mozilla.github.io/nunjucks/ as a client side solution, it is easy to add a server side render solution with quickjs or a nodejs.
`extends file(.ext)` work for me.
Aug 12
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 8/12/21 3:55 AM, workman wrote:
 diet-ng is great and I love to use it,  but it is too slow to compile.
 
 One idea to speedup is use preprocessor like DMD compile C header file 
 with GCC -E.
 
 A diet-ng compiler write in D, translate template file into D source 
 code, then mixin from projects.  A runtime diet-ng compiler will easaly 
 speed up 1000 times compare to CTFE version, and you don't get any 
 performance loss.
 
 A runtime diet-ng compiler also possible compile template into dynamic 
 library file,   you edit view files and a file monitor will recompile 
 the dynamic library then http server reload it, no need recompile full 
 projects.
 
 Add new feature into diet will be more easy, since developer don't limit 
 to use CTFE code.
 
 D CTFE is great, but not suit for task like dieg-ng.
diet-ng is really hard to deal with at runtime. I added a feature that allows any *non-code* portions of the template to be reinterpreted at runtime, so for instance, if you want to add or remove an html class from an element, that is doable without rebuilding. I also created a [diet precompiler](https://code.dlang.org/packages/dietpc) to move most of the CTFE heavy-lifting to a pre-compilation step (it still runs a checksum on the files at compile-time, which is annoying to me, but I haven't bothered yet to strip that part out). But changing the code means recompiling. And that is not always easy, because diet templates are compiled with specific context and aliases, which are not easy to replicate as a subpart of the project. I have considered splitting my routes out into individual subprojects to see if that helps with the build times. With precompiling templates and live mode, diet-ng is at least tolerable for development. If I were to do my own template system, with my experience in diet-ng, it would be focused on runtime interpretation (only a change to the input data structure or backend code would require a recompile) with the ability to compile everything for performance. -Steve
Aug 12