www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.internals - Detect CTFE in AssignExp:semantic

reply Lucia Cojocaru <lucia.mcojocaru gmail.com> writes:
Hello,

Brief context description:
I'm trying to replace calls to druntime functions such as 
_d_arraycopy [0] with calls to templates. These calls are 
generated by the compiler here [1] for array assignments like 
below:

int[2] a = [1, 2];
int[2] b;

b[] = a[]; // _d_array_copy call

There are several such druntime functions and the purpose of this 
is to avoid pulling druntime as a dependency unless necessary. 
The templates don't "exist" if not used and many of them can be 
implemented without pulling any dependency.

--------------------------------------------------------------

I implemented a lowering in AssignExp:semantic to a template call 
for the use case described above. However compilation fails for 
this code (I extracted this bit from druntime code):

   1
   2 char[] mangle(T)(const(char)[] fqn, char[] dst = null)  safe 
pure nothrow
   3 {
   4     import core.internal.string : numDigits, 
unsignedToTempString;
   5
   6     static struct DotSplitter
   7     {
   8          safe pure nothrow:
   9             const(char)[] s;
  10
  11          property bool empty() const { return !s.length; }
  12
  13          property const(char)[] front() const
  14         {
  15             immutable i = indexOfDot();
  16             return i == -1 ? s[0 .. $] : s[0 .. i];
  17         }
  18
  19         void popFront()
  20         {
  21         }
  22
  23         private ptrdiff_t indexOfDot() const
  24         {
  25             foreach (i, c; s) if (c == '.') return i;
  26             return -1;
  27         }
  28     }
  29
  30     size_t len = "_D".length;
  31     foreach (comp; DotSplitter(fqn))
  32         len += numDigits(comp.length) + comp.length;
  33
  34     return ['a', 'b', 'c'];
  35 }
  36
  37 char[] mangleFunc(T:FT*, FT)(const(char)[] fqn, char[] dst = 
null)  safe pure nothrow if (is(FT == function))
  38 {
  39         return mangle!FT(fqn, dst);
  40 }
  41
  42 pragma(mangle, mangleFunc!(int function(int))("a.b"));
  43
  44 int main() {
  45     return 0;
  46 }

The error is:
ctfe.d(28): Error: 2 variable __r57 cannot be read at compile time
ctfe.d(28):        called from here: _d_arraycopyT(this.s[], 
__r57, 1u)
ctfe.d(18):        called from here: this.indexOfDot()
ctfe.d(34):        called from here: __r58.front()
ctfe.d(42):        called from here: mangle(fqn, dst)
ctfe.d(45):        called from here: mangleFunc("a.b", null)

This is probably because of an issue with CTFE and the solution 
may be to avoid generating calls to the template in a CTFE 
context. The code in AssignExp:semantic [2] would look like this:

if (__ctfe)
{
   // add template call to AST
}
else
{
   // old code
}
However I was not able to detect this situation from 
AssignExp:semantic.
1. __ctfe doesn't seem to be set
2. performing similar checks with the ones in dinterpret which 
throw the error [3] don't distinguish this situation
3. checking the scope for ctfe doesn't work either: sc.flags & 
SCOPEctfe

Is there a way to achieve this? Maybe I am looking at this 
problem from a wrong angle.

Thank you,
Lucia

[0]https://github.com/dlang/druntime/blob/2db828bd4f21807254b770b3ec304f14596a9805/src/rt/arraycat.d#L22

[1] https://github.com/dlang/dmd/blob/master/src/e2ir.d#L2702

[2] 
https://github.com/dlang/dmd/blob/master/src/expression.d#L13545

[3] 
https://github.com/dlang/dmd/blob/master/src/dinterpret.d#L2381
Jan 10
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
The way to attack these sorts of problems is to not attempt the complete 
solution as the first step.

Start with a massive simplification - for example, just having mangle() return 
"hello". If that fails in a confusing manner, go even simpler.

Once the very simple solution works, then start adding in the complex solution 
bit by bit, ensuring each piece works before adding the next.
Jan 10
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/10/2017 07:13 AM, Lucia Cojocaru wrote:
 Hello,

 Brief context description:
 I'm trying to replace calls to druntime functions such as _d_arraycopy
 [0] with calls to templates.
[snip] You may also want to push your WIP into your dmd branch on github so people can take a look. Thanks! -- Andrei
Jan 10
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 10 January 2017 at 12:13:18 UTC, Lucia Cojocaru wrote:
 Hello,

 Brief context description:
 I'm trying to replace calls to druntime functions such as 
 _d_arraycopy [0] with calls to templates. These calls are 
 generated by the compiler here [1] for array assignments like 
 below:

 [...]
I cannot say much from this snippet alone. Please discribe what you want to archive concretely. As previously mentioned it would also help to push your code to a visible branch. I will take a look shortly.
Jan 10
parent reply Lucia Cojocaru <lucia.mcojocaru gmail.com> writes:
The dmd code (e2ir.d and expression.d are of interest):
https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090

The druntime template:
https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954

The example which doesn't compile is a standalone code snippet 
which reproduces the error encountered. I started from the code 
in druntime and simplified it to that point. As of your 
suggestions, I will simplify it further and come back with 
another example.

Thanks!
Jan 11
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru 
wrote:
 The dmd code (e2ir.d and expression.d are of interest):
 https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090
Calls for the old dmd<->C-API are very different from template functions calls, e.g. take a look at how _xOpEquals is called. https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496
 The druntime template:
https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954
 The example which doesn't compile is a standalone code snippet 
 which reproduces the error encountered. I started from the code 
 in druntime and simplified it to that point. As of your 
 suggestions, I will simplify it further and come back with 
 another example.
You cannot distinguish between ctfe/non-ctfe during semantic. Only the backend/glue layer differs between CTFE (interpret.d) and IR/codegen (e2ir). If you want to convert a C-API intrinsic to a template function call, you'd usually deal with __ctfe in druntime not in the compiler. DMD will always call the druntime function and if that happens during CTFE, it'll get interpreted.
Jan 11
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/11/17 3:16 PM, Martin Nowak wrote:
 On Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru wrote:
 The dmd code (e2ir.d and expression.d are of interest):
 https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090
Calls for the old dmd<->C-API are very different from template functions calls, e.g. take a look at how _xOpEquals is called. https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496
Cool. That looks different from https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090, is the latter appropriate as well? Or perhaps that's why the error with reading the variable during compilation?
 The druntime template:
https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954
 The example which doesn't compile is a standalone code snippet which
 reproduces the error encountered. I started from the code in druntime
 and simplified it to that point. As of your suggestions, I will
 simplify it further and come back with another example.
You cannot distinguish between ctfe/non-ctfe during semantic. Only the backend/glue layer differs between CTFE (interpret.d) and IR/codegen (e2ir). If you want to convert a C-API intrinsic to a template function call, you'd usually deal with __ctfe in druntime not in the compiler. DMD will always call the druntime function and if that happens during CTFE, it'll get interpreted.
OK, but the problem here is it indicates a problem at the call site of _d_arraycopyT, not inside the implementation. Is there an issue with the way the code is generated? Also, as an aside: the _d_arraycopyT should probably go like this: D _d_arraycopyT(S, D)(S from, D to) { ... } You don't need size because it's from[0].sizeof. Correct? Andrei
Jan 11
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 January 2017 at 17:10:15 UTC, Andrei 
Alexandrescu wrote:
 On 1/11/17 3:16 PM, Martin Nowak wrote:
 On Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru 
 wrote:
 The dmd code (e2ir.d and expression.d are of interest):
 https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090
Calls for the old dmd<->C-API are very different from template functions calls, e.g. take a look at how _xOpEquals is called. https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496
Cool. That looks different from https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090, is the latter appropriate as well? Or perhaps that's why the error with reading the variable during compilation?
 The druntime template:
https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954
 The example which doesn't compile is a standalone code 
 snippet which
 reproduces the error encountered. I started from the code in 
 druntime
 and simplified it to that point. As of your suggestions, I 
 will
 simplify it further and come back with another example.
You cannot distinguish between ctfe/non-ctfe during semantic. Only the backend/glue layer differs between CTFE (interpret.d) and IR/codegen (e2ir). If you want to convert a C-API intrinsic to a template function call, you'd usually deal with __ctfe in druntime not in the compiler. DMD will always call the druntime function and if that happens during CTFE, it'll get interpreted.
OK, but the problem here is it indicates a problem at the call site of _d_arraycopyT, not inside the implementation. Is there an issue with the way the code is generated?
Don't really understand your question. What are the 2 problems you refer to? The difference is fairly simple but huge: - C intrinsics - AssignExp.semantic - e2ir => call RTLSYM_SYM - interpret => special handling - D lowering - AssignExp.semantic lowered to CallExp of object._arrayCopy - normal function call and no special CTFE handling
 Also, as an aside: the _d_arraycopyT should probably go like 
 this:

 D _d_arraycopyT(S, D)(S from, D to) { ... }
 You don't need size because it's from[0].sizeof. Correct?
Just convert the assignment to a function call, the backend deals with optimizations et.al. Also this seems to be used not only for static arrays. NB: - leave aways the _d prefix it's only needed to namespace extern(C) functions with flat mangling - prolly should be _arrayCopy(T)(in T[] from, T[] to) as AssignExp.semantic already takes care of conversions
Jan 11
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/11/17 7:02 PM, Martin Nowak wrote:
 Don't really understand your question. What are the 2 problems you refer
 to?
The current implementation: D _d_arraycopyT(S, D)(S from, D to, uint size) { import core.stdc.string; (() trusted => memcpy(cast(void*)to.ptr, from.ptr, to.length * size))(); return to; } The current error: ctfe.d(28): Error: 2 variable __r57 cannot be read at compile time ctfe.d(28): called from here: _d_arraycopyT(this.s[], __r57, 1u) i.e. NOT in the use of cast, ptr, or memcpy. The error points to the call site. The recommended implementation: D _d_arraycopyT(S, D)(S from, D to, uint size) { if (__ctfe) import core.stdc.string; (() trusted => memcpy(cast(void*)to.ptr, from.ptr, to.length * size))(); } else { foreach (i, ref e; from) to[i] = e; } return to; } It doesn't seem like the recommended implementation will make the error go away.
 The difference is fairly simple but huge:

 - C intrinsics
   - AssignExp.semantic
     - e2ir => call RTLSYM_SYM
     - interpret => special handling

 - D lowering
   - AssignExp.semantic lowered to CallExp of object._arrayCopy
     - normal function call and no special CTFE handling
I'm not familiar with the code, so although yes these are different I don't know how they translate into what needs to be done to make this work.
 Also, as an aside: the _d_arraycopyT should probably go like this:

 D _d_arraycopyT(S, D)(S from, D to) { ... }
 You don't need size because it's from[0].sizeof. Correct?
Just convert the assignment to a function call, the backend deals with optimizations et.al. Also this seems to be used not only for static arrays.
I'm saying the third parameter is entirely redundant in all cases. It's the sizeof the element.
 NB:
   - leave aways the _d prefix it's only needed to namespace extern(C)
 functions with flat mangling
   - prolly should be _arrayCopy(T)(in T[] from, T[] to) as
 AssignExp.semantic already takes care of conversions
Yah, once we're past the hurdle of getting this to basically work these are good touches. Andrei
Jan 11
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 11 January 2017 at 18:59:29 UTC, Andrei 
Alexandrescu wrote:
         foreach (i, ref e; from) to[i] = e;
I would guess foreach(i; auto ref e; from) to[i] = e; is a little more compatible.
Jan 11
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/11/17 8:34 PM, Stefan Koch wrote:
 On Wednesday, 11 January 2017 at 18:59:29 UTC, Andrei Alexandrescu wrote:
         foreach (i, ref e; from) to[i] = e;
I would guess foreach(i; auto ref e; from) to[i] = e; is a little more compatible.
Why? Aren't operands at this point built-in slices? -- Andrei
Jan 11
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 11 January 2017 at 22:14:18 UTC, Andrei 
Alexandrescu wrote:
 On 1/11/17 8:34 PM, Stefan Koch wrote:
 On Wednesday, 11 January 2017 at 18:59:29 UTC, Andrei 
 Alexandrescu wrote:
         foreach (i, ref e; from) to[i] = e;
I would guess foreach(i; auto ref e; from) to[i] = e; is a little more compatible.
Why? Aren't operands at this point built-in slices? -- Andrei
it could still be that you are assigning from a const slice.
Jan 11
parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 January 2017 at 23:44:29 UTC, Stefan Koch wrote:
 it could still be that you are assigning from a const slice.
In which case ref for the foreach parameter still works.
Jan 12
parent reply Lucia Cojocaru <lucia.mcojocaru gmail.com> writes:
On Thursday, 12 January 2017 at 11:09:09 UTC, Martin Nowak wrote:
 On Wednesday, 11 January 2017 at 23:44:29 UTC, Stefan Koch 
 wrote:
 it could still be that you are assigning from a const slice.
In which case ref for the foreach parameter still works.
I updated the code based on your comments. Let me know if this is better. The issue still persists. druntime https://github.com/somzzz/druntime/commit/fb51f34cb0bdd96daaa92ab22773bf93778d4d11 dmd https://github.com/somzzz/dmd/commit/b3cf4625ce733e4b3f219bcd5069d9042430b594
Jan 16
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 01/16/2017 11:22 PM, Lucia Cojocaru wrote:
 druntime
 https://github.com/somzzz/druntime/commit/fb51f34cb0bdd96daaa92ab22773bf93778d4d11
https://github.com/somzzz/druntime/commit/fb51f34cb0bdd96daaa92ab22773bf93778d4d11#commitcomment-20497420
 dmd
 https://github.com/somzzz/dmd/commit/b3cf4625ce733e4b3f219bcd5069d9042430b594
Looks like you're running into some issues with the "magic" temporary variables generated by the compiler. https://github.com/dlang/dmd/blob/a5f823a59d11bb02a56384891eb80a55af467e00/src/statementsem.d#L886-L907 The error is generated here https://github.com/dlang/dmd/blob/c1d138cc3860ecf8cbe06090cc321f4d5502b6ee/src/dinterpret.d#L2441 with hasValue(v) == false meaning that the variable was not interpreted (put on the stack) before usage. I guess this happens because you're actually converting the initialization of the temporary into a function call. T __r154 = slice[]; lowered into: _d_arraycopyT(__r154, slice[]); So the function call gets interpreted before the variable was initialized by CTFE, not exactly sure how to best resolve this. Maybe simply skip lowering ConstructExp (subclass of AssignExp used for construction) for now. You can recognize construction inside of AssignExp by looking at the op, `op == TOKconstruct`. -Martin
Jan 16
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 11 January 2017 at 17:10:15 UTC, Andrei 
Alexandrescu wrote:
 Calls for the old dmd<->C-API are very different from template 
 functions
 calls, e.g. take a look at how _xOpEquals is called.
 https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496
Sorry, it's not the best example, b/c that also generates a function. Added inline comments on https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090 which isn't far off.
 Or perhaps that's why the error with reading the variable 
 during compilation?
Not sure where that comes from, there are unrelated changes as well.
Jan 12
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru 
wrote:
 The dmd code (e2ir.d and expression.d are of interest):
 https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090

 The druntime template:
 https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954

 The example which doesn't compile is a standalone code snippet 
 which reproduces the error encountered. I started from the code 
 in druntime and simplified it to that point. As of your 
 suggestions, I will simplify it further and come back with 
 another example.

 Thanks!
You should not need to special case ctfe inside the compiler for this. Rather the template should have if (__ctfe) inside it if those are needed. However I would advise against splitting code-paths, if it is not _strictly_ necessary. I will be able to provide more help should you need it. Just contact me if you encounter further problems.
Jan 11
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/11/17 4:25 PM, Stefan Koch wrote:
 You should not need to special case ctfe inside the compiler for this.
 Rather the template should have if (__ctfe) inside it if those are needed.
 However I would advise against splitting code-paths, if it is not
 _strictly_ necessary.
That's what confuses me, it's the read of the temporary not the code inside the function. Would branching inside the function help with that? -- Andrei
Jan 11
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 11 January 2017 at 18:51:08 UTC, Andrei 
Alexandrescu wrote:
 On 1/11/17 4:25 PM, Stefan Koch wrote:
 You should not need to special case ctfe inside the compiler 
 for this.
 Rather the template should have if (__ctfe) inside it if those 
 are needed.
 However I would advise against splitting code-paths, if it is 
 not
 _strictly_ necessary.
That's what confuses me, it's the read of the temporary not the code inside the function. Would branching inside the function help with that? -- Andrei
Depending on how you choose optimize the certain cases. You e.g. using sse2 instructions for copies of 128bit values, It may become non-ctfeable. Then you need a ctfeable branch in order for the function to still work at ctfe.
Jan 11
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/11/2017 10:51 AM, Andrei Alexandrescu wrote:
 That's what confuses me, it's the read of the temporary not the code inside the
 function. Would branching inside the function help with that? -- Andrei
The example is about 40 lines of code. The way to figure it out is to simplify the example. For starters, replace: char[] mangle(T)(const(char)[] fqn, char[] dst = null) with: char[] mangle(const(char)[] fqn) Note that the template doesn't even use T or dst. Nor does it use unsignedToTempString. Replace numDigits() with 1. Rewrite the foreach loop to use primitives, then reduce the primitives (like removing popFront() which does nothing). Never going to figure out what is wrong if the example is larded up with obfuscation and distraction. I bet it will shrink down to 3 lines that exhibit the same issue, at which point the problem will be obvious.
Jan 11
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Take a look at the D test suite, such as:

     https://github.com/dlang/dmd/blob/master/test/runnable/test42.d

Each of these tests cases started out as some long complicated thing that got 
reduced to a very small self-contained test case.
Jan 11
parent reply Lucia Cojocaru <lucia.mcojocaru gmail.com> writes:
On Thursday, 12 January 2017 at 01:29:12 UTC, Walter Bright wrote:
 Take a look at the D test suite, such as:

     
 https://github.com/dlang/dmd/blob/master/test/runnable/test42.d

 Each of these tests cases started out as some long complicated 
 thing that got reduced to a very small self-contained test case.
Here is a smaller example: ptrdiff_t f(char[] s) { // foreach(c; s) if (c == '.') return 0; // (0)fails with the same error message foreach(char c; s) if (c == '.') return 0; // (1) // foreach(dchar c; s) if (c == '.') return 0; // (2) different compile path, doesn't fail /* for (size_t i = 0; i < s.length; i++) { if (s[i] == '.') return i; } */ return -1; } pragma(msg, f(['z', 'b', 'c', '.', 'd'])); ------------------------------------------------------- ctfe.d(5): Error: variable __r54 cannot be read at compile time ctfe.d(5): called from here: _d_arraycopyT(__r54, s[]) ctfe.d(15): called from here: f(['z', 'b', 'c', '.', 'd']) ctfe.d(15): while evaluating pragma(msg, f(['z', 'b', 'c', '.', 'd'])) ------------------------------------------------------- It looks like the problem is with the foreach loop. In the example above, the (0) and (1) loops result in the compilation error mentioned. Loop (2) with dchar is fine. I also tried using a regular for loop and this one compiled.
Jan 16
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/16/2017 2:09 PM, Lucia Cojocaru wrote:
 ptrdiff_t f(char[] s) {
     // foreach(c; s) if (c == '.') return 0; // (0)fails with the same error
 message
     foreach(char c; s) if (c == '.') return 0; // (1)
     // foreach(dchar c; s) if (c == '.') return 0; // (2) different compile
 path, doesn't fail

     /*
     for (size_t i = 0; i < s.length; i++)
     {
         if (s[i] == '.') return i;
     }
     */
     return -1;
 }

 pragma(msg, f(['z', 'b', 'c', '.', 'd']));
I just compiled it with: dmd -c bug8 and it prints: 0 with no error message.
Jan 16
parent reply Martin Nowak <code dawg.eu> writes:
On Tuesday, 17 January 2017 at 03:32:41 UTC, Walter Bright wrote:
 I just compiled it with:
The problem comes from also doing the lowering for initialization, hence the CTFE interpreter has never seen the variable value before it is used. http://forum.dlang.org/post/o5jrti$n0t$1 digitalmars.com
Jan 17
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2017 4:39 AM, Martin Nowak wrote:
 The problem comes from also doing the lowering for initialization, hence the
 CTFE interpreter has never seen the variable value before it is used.
 http://forum.dlang.org/post/o5jrti$n0t$1 digitalmars.com
Thanks!
Jan 17