www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Is it possible to "cache" results of compile-time executions between

reply TheFlyingFiddle <kurtyan student.chalmers.se> writes:
Context:
I am currently writing a small library that compiles sql strings 
at compile-time and generates query objects.

Something like this:

unittest
{
     mixin Sql!(q{
        select feed.url, feed.title
        from users
        join user_feeds as feed
        on users.id = feed.user
        where user.id = {user}
     }, DBInterface) UserFeeds;

     UserFeeds.Query query;
     query.user = 1; //some user

     auto con   = //Open db connection
     auto feeds = con.execute(query);
     foreach(f; feeds)
     {
         writef("Feed: %s\n\tTitle: %s\n", f.url, f.title);
     }
}

The parsing step does some amount work like validates the sql 
query, makes sure that tables "user" and "user_feed" exists in 
DBInterface and creates a query having "user" as input and a 
result type containing url, title with appropriate type.

Now the compile times for parsing a small number of queries are 
marginal. However, as the queries become more numerous and 
complex compile-times starts to become a problem.

Currently I "solve"tm long compile times by having the 
DBInterface and queries be compiled into a separate lib and 
linking to that lib from the main application.

Having a dedicated db lib fixes compile times for the main 
application. But I am wondering if it's possible to work on a 
lower level of granularity. Somehow "cache" the queries between 
compiles, without resorting to each query being compiled into 
it's own lib/object file.

Does D have any facilities that could make this possible?
Jan 24
parent reply TheFlyingFiddle <kurtyan student.chalmers.se> writes:
On Tuesday, 24 January 2017 at 11:19:58 UTC, TheFlyingFiddle 
wrote:
 Does D have any facilities that could make this possible?
It seems that there is a feature I was unaware of/forgot called Import Expressions. unittest { enum s = import("myfile"); } Is there something similar to this for outputting files at compile-time? Or do I need to keep the source around and write it at run-time?
Jan 24
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tuesday, 24 January 2017 at 12:14:05 UTC, TheFlyingFiddle 
wrote:
 unittest
 {
    enum s = import("myfile");
 }

 Is there something similar to this for outputting files at 
 compile-time?
no. this is by design, so it won't be fixed. sorry. you may use build script that will create the code first, and you can dump `pragma(msg, …);` to file, but that's all.
Jan 24
next sibling parent TheFlyingFiddle <kurtyan student.chalmers.se> writes:
On Tuesday, 24 January 2017 at 12:19:33 UTC, ketmar wrote:
 On Tuesday, 24 January 2017 at 12:14:05 UTC, TheFlyingFiddle 
 wrote:
 unittest
 {
    enum s = import("myfile");
 }

 Is there something similar to this for outputting files at 
 compile-time?
no. this is by design, so it won't be fixed. sorry. you may use build script that will create the code first, and you can dump `pragma(msg, …);` to file, but that's all.
Yeah, I guess allowing the compiler to produce arbitrary files would be problematic from a security standpoint. Thanks for the pragma idea! Wrapping the build in a script is a satisfactory solution for me.
Jan 24
prev sibling parent reply TheFlyingFiddle <kurtyan student.chalmers.se> writes:
On Tuesday, 24 January 2017 at 12:19:33 UTC, ketmar wrote:
 On Tuesday, 24 January 2017 at 12:14:05 UTC, TheFlyingFiddle 
 wrote:
 unittest
 {
    enum s = import("myfile");
 }

 Is there something similar to this for outputting files at 
 compile-time?
no. this is by design, so it won't be fixed. sorry. you may use build script that will create the code first, and you can dump `pragma(msg, …);` to file, but that's all.
Thanks again. Wrapping dmd with a script worked wonders! Now i'm able to do this: From the old: (a) unittest { mixin Sql!(...); mixin Sql!(...); ... mixin Sql!(...); } To the new: (b) unittest { mixin Cache!(Sql, ...); mixin Cache!(Sql, ...); ... mixin Cache!(Sql, ...); } For (a) the build times are in the 10-30s always For (b) the build times are in the 10-30s the first time and subseconds later. Each query just adds a few ms to the build time now! Additionally even if dmd crashes with an out of memory exception (which still happens with the current ctfe engine) most of the queries will have already been built and dmd can be restarted. After the restart the built queries are loaded via caching and dmd can finish working on the leftovers. Everything turned out soooo much better than expected :)
Jan 24
parent reply TheFlyingFiddle <kurtyan student.chalmers.se> writes:
On Tuesday, 24 January 2017 at 16:41:13 UTC, TheFlyingFiddle 
wrote:
 Everything turned out soooo much better than expected :)
Added bonus is that mixin output can be viewed in the generated files :D
Jan 24
parent reply Profile Anaysis <PA gotacha.com> writes:
On Tuesday, 24 January 2017 at 16:49:03 UTC, TheFlyingFiddle 
wrote:
 On Tuesday, 24 January 2017 at 16:41:13 UTC, TheFlyingFiddle 
 wrote:
 Everything turned out soooo much better than expected :)
Added bonus is that mixin output can be viewed in the generated files :D
Could you post your solution? I suggest we get a real caching module like above that has the extra feature of hashing the mixin strings. This way the caching mechanism can validate if the mixin strings have changed. Put the hash in a comment in the output file that used to test if the input string has the same hash. If it does, simply use the output file, else, regenerate. Adds some overhead but keeps things consistent. (Since I'm not sure what Cache!() is, I'm assuming it doesn't do this)
Jan 24
next sibling parent reply Profile Anaysis <PA gotacha.com> writes:
On Tuesday, 24 January 2017 at 21:36:50 UTC, Profile Anaysis 
wrote:
...
Maybe with all this talk of the new CTFE engine being developed, a similar mechanism can be used optionally? This could help with debugging also. In debug mode, the cfte mixin's are written to disk with hash, if they are not a string themselves. (could be done with all cfte's, I suppose, but not sure about performance and consistency) Then debuggers can use the outputed cfte's for proper analysis, line breaking, etc...
Jan 24
parent TheFlyingFiddle <kurtyan student.chalmers.se> writes:
On Tuesday, 24 January 2017 at 21:41:12 UTC, Profile Anaysis 
wrote:
 On Tuesday, 24 January 2017 at 21:36:50 UTC, Profile Anaysis 
 wrote:
...
Maybe with all this talk of the new CTFE engine being developed, a similar mechanism can be used optionally? This could help with debugging also. In debug mode, the cfte mixin's are written to disk with hash, if they are not a string themselves. (could be done with all cfte's, I suppose, but not sure about performance and consistency) Then debuggers can use the outputed cfte's for proper analysis, line breaking, etc...
Would be nice to have something like this in dmd. Would be even better if it could work on templates as well. No more stepping though functions filled with static if :). I think it would be possible to make something like this work to: template Generator(T...) { string Generator() { //Complex ctfe with T... that generates strings and nothing else. } } If it's possible to quickly detect changes to all T arguments one could cache things on this form to. For example: mixin Cache!("some_file_name", Generator, size_t, int, 2, "hello", MyStruct(3), MyType); The problem would be detecting changes in the arguments. As long as one is able to get a unique hash from each input element it should work fine I think. I guess it would be required to reflect over the members of the structs/classes to lookup attributes and such. If the generation stage is time consuming this might be worth it... But it's not gonna be "almost free" like for DSLs. Basic types (not including functions/delegates don't see how this could work without writing mixin code targeted at caching) should be simple/trivial to detect changes to however.
Jan 24
prev sibling parent TheFlyingFiddle <kurtyan student.chalmers.se> writes:
On Tuesday, 24 January 2017 at 21:36:50 UTC, Profile Anaysis 
wrote:
 On Tuesday, 24 January 2017 at 16:49:03 UTC, TheFlyingFiddle 
 wrote:
 On Tuesday, 24 January 2017 at 16:41:13 UTC, TheFlyingFiddle 
 wrote:
 Everything turned out soooo much better than expected :)
Added bonus is that mixin output can be viewed in the generated files :D
Could you post your solution? I suggest we get a real caching module like above that has the extra feature of hashing the mixin strings. This way the caching mechanism can validate if the mixin strings have changed. Put the hash in a comment in the output file that used to test if the input string has the same hash. If it does, simply use the output file, else, regenerate. Adds some overhead but keeps things consistent. (Since I'm not sure what Cache!() is, I'm assuming it doesn't do this)
This is the solution I through together: // module mixin_cache.d mixin template Cache(alias GenSource, string from, string path) { import core.internal.hash; import std.conv : to; //Hash to keep track of changes enum h = hashOf(from); enum p = path ~ h.to!string ~ ".txt"; //Check if the file exists else suppress errors //The -J flag needs to be set on dmd else //this always fails static if(__traits(compiles, import(p))) { //Tell the wrapper that we loaded the file p //_importing is a magic string pragma(msg, "_importing"); pragma(msg, p); mixin(import(p)); } else { //We don't have a cached file so generate it private enum src = GenSource!(from); static if(__traits(compiles, () { mixin(src); })) { //_exporing_start_ tells the wrapper to begin //outputing the generated source into file p pragma(msg, "_exported_start_"); pragma(msg, p); pragma(msg, src); pragma(msg, "_exported_end_"); } mixin(src); } } To make this work I wrap dmd in a d script like this: (ignoring some details as what i've got is not really tested yet) // dmd_with_mixin_cache.d void main(string[] args) { auto dmdargs = ... //Fix args etc. auto dmd = pipeProcess(dmdargs, Redirect.stderr); foreach(line; dmd.stderr.byLine(KeepTerminator.yes)) { if(line.startsWith("_exported_start_")) { //Parse file and store source in a file //Keep going until _exported_end_ } else if(line.startsWith("_importing")) { //A user imported a file. (don't delete it!) } else { //Other output from dmd like errors / other pragma(msg, ...) } } //Files not imported / exported could be stale //delete them. Unless we got a compile error from dmd //Then don't delete anything. } The cache template and the small wrapper that wraps dmd was all that were needed. usage is somthing like this: template Generate(string source) { string Generate() { //Do some complex ctfe here //can't wait for the new ctfe engine! foreach(i; 0 .. 100_000) { } return source; } } mixin Cache!("Some interesting DSL or similar", __MODULE__); //some_filename is not really needed but can be nice when browsing the //mixin code to see where it came from. (approximately anwyays) This is it. If you want I can post a full solution sometime later this week but I want to clean up what I have first.
Jan 24