• Manu via Digitalmars-d (10/10) Oct 01 2016 This comes up a lot.
• Stefan Koch (9/19) Oct 01 2016 We should create a file where the string-mixins are expanded, I
• Andrei Alexandrescu (5/7) Oct 02 2016 Yes, Stefan it would be terrific if you could keep an eye on it while
• Stefan Koch (14/18) Oct 03 2016 instantiation ?
• Ethan Watson (7/10) Oct 03 2016 You should be able to do everything with the example code you
• Andrei Alexandrescu (4/20) Oct 03 2016 Understood, then keep this on the back burner and forge ahead with the
• Ethan Watson (8/10) Oct 02 2016 Further to this. I tried generating .di files the other day for
• Manu via Digitalmars-d (3/13) Oct 02 2016 That's genius! :)
• Andrei Alexandrescu (2/18) Oct 02 2016 Yah, very very interesting idea. -- Andrei
• Jonathan Marler (7/17) Oct 03 2016 Yes, having the mixins expanded without the surrounding code
• Stefan Koch (3/11) Oct 03 2016 Oh yes. it is not that more much work :)
• Manu via Digitalmars-d (10/22) Oct 03 2016 What case of template instantiation where there are no mixins involved
• Stefan Koch (7/40) Oct 03 2016 static ifs are resolved when the compiler sees the
• Stefan Koch (7/28) Oct 04 2016 A small update on this.
• ZombineDev (13/41) Oct 05 2016 The simplest workaround is to run recursion heavy code in a fiber

Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:

This comes up a lot.
As far as I know, it's not solved. What shall we do?

I feel like a simple solution would be to have the compiler emit a
<filename>_mixin.d file populated with all the mixin expansions beside
the .obj files, and have the debuginfo refer to that fabricated source
file?

It might look a little bit weird jumping into code where the
surrounding scope is not visible (unless that were copied over too?),
but it's better than what we have now.

Are there any other commonly proposed solutions?

Stefan Koch <uplink.coder googlemail.com> writes:

On Sunday, 2 October 2016 at 03:36:31 UTC, Manu wrote:
 This comes up a lot.
 As far as I know, it's not solved. What shall we do?

 I feel like a simple solution would be to have the compiler 
 emit a <filename>_mixin.d file populated with all the mixin 
 expansions beside the .obj files, and have the debuginfo refer 
 to that fabricated source file?

 It might look a little bit weird jumping into code where the 
 surrounding scope is not visible (unless that were copied over 
 too?), but it's better than what we have now.

 Are there any other commonly proposed solutions?

We should create a file where the string-mixins are expanded, I 
agree.
However I am not sure if this is a quick-fix or a more tricky 
thing.

In theory the PrettyPrinter we have should give as a way to 
produce such a file.

I have been meaning to work on this issue for a while now, alas 
there is always something else to do :)

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/02/2016 01:00 AM, Stefan Koch wrote:
 We should create a file where the string-mixins are expanded, I agree.
 However I am not sure if this is a quick-fix or a more tricky thing.

Yes, Stefan it would be terrific if you could keep an eye on it while 
working on the engine. A file with properly handed dependencies could 
serve as instantiation cache and save a ton of re-instantiation waste. 
Thanks! -- Andrei

Stefan Koch <uplink.coder googlemail.com> writes:

On Sunday, 2 October 2016 at 12:27:23 UTC, Andrei Alexandrescu 
wrote:
 Yes, Stefan it would be terrific if you could keep an eye on it 
 while working on the engine. A file with properly handed 
 dependencies could serve as instantiation cache and save a ton 
 of re-instantiation waste. Thanks! -- Andrei

instantiation ?
Do you mean mixin-expansion ?
or are you talking about mixins created in templates ?

Although Caching mixin expansions hits on the same problem as 
chaching template-instances, it is more difficult since a mixin 
can appear anywhere.
and one has to worry about a much wider scope.

I'll see what I can come up with, for now the debugging is 
priority.
 Ethan can you share code illustrating your usecase.
(My head is inside compiler internals and building test-code is a 
rather unpleasant context switch)

Ethan Watson <gooberman gmail.com> writes:

On Monday, 3 October 2016 at 11:42:25 UTC, Stefan Koch wrote:
  Ethan can you share code illustrating your usecase.
 (My head is inside compiler internals and building test-code is 
 a rather unpleasant context switch)

You should be able to do everything with the example code you 
have. My intention there is to compile everything in the 
acorelibrary module in to a library and statically link against 
that. As such, you'll want to do the .di generation on those 
files. The behaviour right now is that all the Binderoo mixins 
will not expand.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/03/2016 07:42 AM, Stefan Koch wrote:
 On Sunday, 2 October 2016 at 12:27:23 UTC, Andrei Alexandrescu wrote:
 Yes, Stefan it would be terrific if you could keep an eye on it while
 working on the engine. A file with properly handed dependencies could
 serve as instantiation cache and save a ton of re-instantiation waste.
 Thanks! -- Andrei

 instantiation ?
 Do you mean mixin-expansion ?
 or are you talking about mixins created in templates ?

Mixin expansions.

 Although Caching mixin expansions hits on the same problem as chaching
 template-instances, it is more difficult since a mixin can appear anywhere.
 and one has to worry about a much wider scope.

 I'll see what I can come up with, for now the debugging is priority.
  Ethan can you share code illustrating your usecase.
 (My head is inside compiler internals and building test-code is a rather
 unpleasant context switch)

Understood, then keep this on the back burner and forge ahead with the 
warm cache. Thanks! -- Andrei

Ethan Watson <gooberman gmail.com> writes:

On Sunday, 2 October 2016 at 05:00:07 UTC, Stefan Koch wrote:
 We should create a file where the string-mixins are expanded, I 
 agree.

Further to this. I tried generating .di files the other day for 
code based on Binderoo. None of the mixins were expanded, which 
resulted in the compiler needing to do all that work again anyway 
and negating the effect of a precompiled library.

If you get to doing that work, it would be fab if you could apply 
it to .di generation. If not as the default, then at least as a 
switch I can provide on the command line.

Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:

On 3 October 2016 at 00:08, Ethan Watson via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 2 October 2016 at 05:00:07 UTC, Stefan Koch wrote:
 We should create a file where the string-mixins are expanded, I agree.


 Further to this. I tried generating .di files the other day for code based
 on Binderoo. None of the mixins were expanded, which resulted in the
 compiler needing to do all that work again anyway and negating the effect of
 a precompiled library.

 If you get to doing that work, it would be fab if you could apply it to .di
 generation. If not as the default, then at least as a switch I can provide
 on the command line.

That's genius! :)

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 10/02/2016 10:44 PM, Manu via Digitalmars-d wrote:
 On 3 October 2016 at 00:08, Ethan Watson via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Sunday, 2 October 2016 at 05:00:07 UTC, Stefan Koch wrote:
 We should create a file where the string-mixins are expanded, I agree.


 Further to this. I tried generating .di files the other day for code based
 on Binderoo. None of the mixins were expanded, which resulted in the
 compiler needing to do all that work again anyway and negating the effect of
 a precompiled library.

 If you get to doing that work, it would be fab if you could apply it to .di
 generation. If not as the default, then at least as a switch I can provide
 on the command line.

 That's genius! :)

Yah, very very interesting idea. -- Andrei

Jonathan Marler <johnnymarler gmail.com> writes:

On Sunday, 2 October 2016 at 03:36:31 UTC, Manu wrote:
 This comes up a lot.
 As far as I know, it's not solved. What shall we do?

 I feel like a simple solution would be to have the compiler 
 emit a <filename>_mixin.d file populated with all the mixin 
 expansions beside the .obj files, and have the debuginfo refer 
 to that fabricated source file?

 It might look a little bit weird jumping into code where the 
 surrounding scope is not visible (unless that were copied over 
 too?), but it's better than what we have now.

 Are there any other commonly proposed solutions?

Yes, having the mixins expanded without the surrounding code 
would make it difficult to debug in some cases.  Maybe generating 
the entire source with the expanded mixins is another option?

mycode.d
obj/mycode_processed.d

Maybe this idea could also be expanded to template instantiation?

Stefan Koch <uplink.coder googlemail.com> writes:

On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler wrote:

 Yes, having the mixins expanded without the surrounding code 
 would make it difficult to debug in some cases.  Maybe 
 generating the entire source with the expanded mixins is 
 another option?

 mycode.d
 obj/mycode_processed.d

That was my intention.
 Maybe this idea could also be expanded to template 
 instantiation?

  Oh yes. it is not that more much work :)

Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:

On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler wrote:

 Yes, having the mixins expanded without the surrounding code would make it
 difficult to debug in some cases.  Maybe generating the entire source with
 the expanded mixins is another option?

 mycode.d
 obj/mycode_processed.d

 That was my intention.
 Maybe this idea could also be expanded to template instantiation?

  Oh yes. it is not that more much work :)

What case of template instantiation where there are no mixins involved
would this make significantly simpler to debug? (I don't know this is
a critical debugability problem as it is...)
Do you mean just substituting 'T' with actual types? resolving static
if's? Hard to know what it should do...

Actually, one case that often bites me is static-foreach unrolling.
That's borderline impossible to debug.
foreach(m; __traits(allMembers,T)) is the classic impossible to debug case.

Stefan Koch <uplink.coder googlemail.com> writes:

On Tuesday, 4 October 2016 at 01:20:01 UTC, Manu wrote:
 On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler 
 wrote:

 Yes, having the mixins expanded without the surrounding code 
 would make it difficult to debug in some cases.  Maybe 
 generating the entire source with the expanded mixins is 
 another option?

 mycode.d
 obj/mycode_processed.d

 That was my intention.
 Maybe this idea could also be expanded to template 
 instantiation?

  Oh yes. it is not that more much work :)

 What case of template instantiation where there are no mixins 
 involved
 would this make significantly simpler to debug? (I don't know 
 this is
 a critical debugability problem as it is...)
 Do you mean just substituting 'T' with actual types? resolving 
 static
 if's? Hard to know what it should do...

 Actually, one case that often bites me is static-foreach 
 unrolling.
 That's borderline impossible to debug.
 foreach(m; __traits(allMembers,T)) is the classic impossible to 
 debug case.

static ifs are resolved when the compiler sees the 
template-instance in semantic3.
And that makes a huge difference in some cases where a template 
is generated by a string-mixin for example.

is not that big of a deal to print out unrolled static foreach.
(as in I can implement in the compiler within 2 days)

Stefan Koch <uplink.coder googlemail.com> writes:

On Tuesday, 4 October 2016 at 01:59:11 UTC, Stefan Koch wrote:
 On Tuesday, 4 October 2016 at 01:20:01 UTC, Manu wrote:
 On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler 
 wrote:

 Yes, having the mixins expanded without the surrounding code 
 would make it difficult to debug in some cases.  Maybe 
 generating the entire source with the expanded mixins is 
 another option?

 mycode.d
 obj/mycode_processed.d

 That was my intention.
 Maybe this idea could also be expanded to template 
 instantiation?

  Oh yes. it is not that more much work :)



A small update on this.
The POC works rather well ...
Except for cases of massive template recursion. (binderoo and 
most of std.traits)
In such cases a stack overflow occurs inside the prettyPrinter.

I am trying to find a work-around.

ZombineDev <petar.p.kirov gmail.com> writes:

On Wednesday, 5 October 2016 at 02:45:53 UTC, Stefan Koch wrote:
 On Tuesday, 4 October 2016 at 01:59:11 UTC, Stefan Koch wrote:
 On Tuesday, 4 October 2016 at 01:20:01 UTC, Manu wrote:
 On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler 
 wrote:

 Yes, having the mixins expanded without the surrounding 
 code would make it difficult to debug in some cases.  Maybe 
 generating the entire source with the expanded mixins is 
 another option?

 mycode.d
 obj/mycode_processed.d

 That was my intention.
 Maybe this idea could also be expanded to template 
 instantiation?

  Oh yes. it is not that more much work :)



 A small update on this.
 The POC works rather well ...
 Except for cases of massive template recursion. (binderoo and 
 most of std.traits)
 In such cases a stack overflow occurs inside the prettyPrinter.

 I am trying to find a work-around.

The simplest workaround is to run recursion heavy code in a fiber 
with large stack size (e.g. 64MB).

// before
auto newExpr = expr.ctfeInterpret();  // some recursion heavy code

// after	
Expression newExpr;
	
import core.thread : Fiber;	
new Fiber(
{
      newExpr = expr.ctfeInterpret();  // some recursion heavy code
}, 64 * 1024 * 1024).call();