www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - template statistics

reply Walter Bright <newshound2 digitalmars.com> writes:
Many people are trying to figure out what templates are blowing up their 
compile. This should help:

https://github.com/dlang/dmd/pull/11208
Jun 01
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Nice! Wishlist: if you could dump out a template instantiation backtrace combined with system time for every template declaration start and end, it should be easy to convert this into callgrind format ( https://valgrind.org/docs/manual/cl-format.html ), which would let us use graphical profilers with it.
Jun 02
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:
 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Nice! Wishlist: if you could dump out a template instantiation backtrace combined with system time for every template declaration start and end, it should be easy to convert this into callgrind format ( https://valgrind.org/docs/manual/cl-format.html ), which would let us use graphical profilers with it.
Ah I didn't know the callgrind format was documented :) I can give you that I already have something similar for my own profiling.
Jun 02
prev sibling next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:
 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Nice! Wishlist: if you could dump out a template instantiation backtrace combined with system time for every template declaration start and end, it should be easy to convert this into callgrind format ( https://valgrind.org/docs/manual/cl-format.html ), which would let us use graphical profilers with it.
Check this branch out. https://github.com/UplinkCoder/dmd/tree/dmd_tracing_2092 It doesn't do the cl-format. But it should help in showing how-to add such per-symbol performance info
Jun 02
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:
 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Nice! Wishlist: if you could dump out a template instantiation backtrace combined with system time for every template declaration start and end, it should be easy to convert this into callgrind format ( https://valgrind.org/docs/manual/cl-format.html ), which would let us use graphical profilers with it.
Update: I've hacked up an extremely basic callgrind dump, just to test the format. It's based on line numbers, not functions, which callgrind doesn't like: "functions" are "file_filename_line_linenr". https://github.com/FeepingCreature/dmd/commit/aa379ba7750b8d32f359 6dc93d3c381cfa90e45 Note I'm *really* not sure if I'm doing the inclusive/exclusive accounting correctly. Did some internal testing. format!() seems to eat up a lot (40%?), not sure if I'm reading this right, but it wouldn't surprise me.
Jun 03
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 08:40:18 UTC, FeepingCreature wrote:
 On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:
 [...]
Update: I've hacked up an extremely basic callgrind dump, just to test the format. It's based on line numbers, not functions, which callgrind doesn't like: "functions" are "file_filename_line_linenr". https://github.com/FeepingCreature/dmd/commit/aa379ba7750b8d32f359 6dc93d3c381cfa90e45 Note I'm *really* not sure if I'm doing the inclusive/exclusive accounting correctly. Did some internal testing. format!() seems to eat up a lot (40%?), not sure if I'm reading this right, but it wouldn't surprise me.
Looks correct. Format is awful.
Jun 03
prev sibling next sibling parent reply Ethan <gooberman gmail.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Oh I have *got* to try this out on my code. (Just need to find the time right now...)
Jun 02
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/2/2020 5:53 AM, Ethan wrote:
 Oh I have *got* to try this out on my code.
I tried it on `writeln("hello");` and the number of templates instantiated seems excessive. It should be investigated. Number Unique Name 46 38 MAKEINTRESOURCE_T(ushort i) 38 0 OldAlias(T) if (!isAggregateType!T || is(Unqual!T == T)) 7 6 staticIndexOf(T, TList...) 1 1 Impl(T) 7 5 isSomeChar(T) 2 1 StringTypeOf(T) 1 1 Impl(T) 8 2 mostNegative(T) if (isNumeric!T || isSomeChar!T || isBoolean!T) 1 1 encode(Flag useReplacementDchar = No.useReplacementDchar)(out char[4] buf, dchar c) 7 7 ModifyTypePreservingTQ(alias Modifier, T) 2 1 Unqual(T) 1 1 put(A)(A writeme) if ((isSomeChar!(Unqual!(ElementType!A)) || is(ElementType!A : const(ubyte))) && i sInputRange!A && !isInfinite!A) 1 1 decodeImpl(bool canIndex, Flag useReplacementDchar = No.useReplacementDchar, S)(auto ref S str, ref size_t index) if (is(S : const(char[])) || isInputRange!S && is(Unqual!(ElementEncodingType!S) == char)) 3 1 isCallable(T...) if (T.length == 1) 1 1 trustedFwrite(T)(FILE* f, const T[] obj) 2 1 _utfException(Flag useReplacementDchar)(string msg, dchar c) 1 1 popFront(C)(ref C[] str) if (Autodecoding && isNarrowString!(C[])) 2 2 IntegralTypeOf(T) 1 1 Impl(T) 1 1 Impl(T) 20 20 genericIndexOf(args...) if (args.length >= 1) 1 1 safeOp(T0, T1)(auto ref T0 a, auto ref T1 b) 5 1 ElementEncodingType(R) 1 1 errnoEnforce(T, string file = __FILE__, uint line = __LINE__)(T value, lazy string msg = null) 1 1 Impl(T) 2 2 put(C)(C c) if (isSomeChar!C || is(C : const(ubyte))) 2 1 exception(S)(S str, string msg) 6 1 isNarrowString(T) 2 1 at(R)(R[] r, size_t i) 1 1 FunctionTypeOf(func...) if (func.length == 1 && isCallable!func) 1 1 MinType(T...) if (T.length >= 1) 5 0 AliasSeq(TList...) 4 1 opDispatch(string name) 1 1 min(T...)(T args) if (T.length >= 2) 8 8 sndAlias(char c0, char c1) 1 1 __equals(T1, T2)(T1[] lhs, T2[] rhs) 55 13 isAggregateType(T) 8 2 isNumeric(T) 7 7 OriginalType(T) 4 1 isInputRange(R) 1 1 empty(T)(in T[] a) 6 3 ElementType(R) 36 11 expectType(T) 24 7 Unqual(T) 1 1 MAKEINTATOM_T()(int i) 1 1 writeln(T...)(T args) 1 1 Impl(T) 3 1 isSomeString(T) 1 1 isSomeFunction(T...) if (T.length == 1) 15 2 TypeDef(T) 1 1 isInfinite(R) 3 1 isStaticArray(T) 18 18 isSame(ab...) if (ab.length == 2) 2 1 front(T)(T[] a) if (Autodecoding && isNarrowString!(T[])) 1 1 decode(Flag useReplacementDchar = No.useReplacementDchar, S)(auto ref S str, ref size_t index) if (i sSomeString!S) 1 1 codeUnitLimit(S) if (isSomeChar!(ElementEncodingType!S)) 2 1 Parameters(func...) if (func.length == 1 && isCallable!func) 1 1 fd_set_custom(uint SETSIZE) 4 4 mmioFOURCC(char c0, char c1, char c2, char c3) 2 1 Demangle(T) 4 3 Flag(string name) 1 1 safeOp(string S) if (S == "<" || S == ">" || S == "<=" || S == ">=" || S == "==" || S == "!=") 2 2 isIntegral(T) 1 1 Impl(T) 5 5 CharTypeOf(T)
Jun 02
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 02:32:54 UTC, Walter Bright wrote:
       46       38   MAKEINTRESOURCE_T(ushort i)
This is a ported C macro in the Windows headers; a fixed cost (as in it won't grow with your program). But it also doesn't have to exist - it could just be a CTFE function, or even just written out inline (all it does is cast a value to a different type!)
       38        0   OldAlias(T) if (!isAggregateType!T || 
 is(Unqual!T == T))
Now this one is a private template inside std.traits and I honestly don't understand why it exists. I suspect it was to paper over a compiler bug in the past and probably is now just unchanged out of fear of breakage. Most the rest are reflection helpers in Phobos, many of which could prolly be trivially replaced with inline checks but .... maybe not a big deal anyway. idk.
Jun 02
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/2/2020 7:41 PM, Adam D. Ruppe wrote:
 Most the rest are reflection helpers in Phobos, many of which could prolly be 
 trivially replaced with inline checks but .... maybe not a big deal anyway.
idk.
The thing is, `writeln("hello")` can be expanded to `core.stdc.stdio.puts("hello")`. Done. Phobos seems to do a lot of "going around the Horn" instead of taking the canal.
Jun 02
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:
 On 6/2/2020 7:41 PM, Adam D. Ruppe wrote:
 Most the rest are reflection helpers in Phobos, many of which 
 could prolly be trivially replaced with inline checks but .... 
 maybe not a big deal anyway. idk.
The thing is, `writeln("hello")` can be expanded to `core.stdc.stdio.puts("hello")`. Done. Phobos seems to do a lot of "going around the Horn" instead of taking the canal.
I have been saying that for years. Apparently you never used `-vcg-ast`.
Jun 02
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 04:46:54 UTC, Stefan Koch wrote:
 On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:
 On 6/2/2020 7:41 PM, Adam D. Ruppe wrote:
 Most the rest are reflection helpers in Phobos, many of which 
 could prolly be trivially replaced with inline checks but 
 .... maybe not a big deal anyway. idk.
The thing is, `writeln("hello")` can be expanded to `core.stdc.stdio.puts("hello")`. Done. Phobos seems to do a lot of "going around the Horn" instead of taking the canal.
I have been saying that for years. Apparently you never used `-vcg-ast`.
I've recently been informed that this may have sounded dismissive. It was not meant that way. I am just a little frustrated that my work to make this issue visible seems to have gone unnoticed. Then again .... I didn't exactly advertise it.
Jun 04
prev sibling next sibling parent reply Nick Treleaven <nick geany.org> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:
 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.
How does writeln tell that a zero terminated string has been passed?
Jun 03
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 1:06 AM, Nick Treleaven wrote:
 How does writeln tell that a zero terminated string has been passed?
D string literals always have a 0 terminator.
Jun 03
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 3:59 PM, Walter Bright wrote:
 On 6/3/2020 1:06 AM, Nick Treleaven wrote:
 How does writeln tell that a zero terminated string has been passed?
D string literals always have a 0 terminator.
All compile-time strings should be both 0 terminated and implicitly convertible to immutable(char)*. I think there are some cases that aren't, but I don't remember all of them. But the question above is still unanswered -- writeln receives a string as an immutable(char)[], and doesn't know that it's from a literal. For example, if writeln blindly assumed that all immutable(char)[] were string literals, that it could pass to puts, this would be a problem: writeln(somestr.idup); -Steve
Jun 03
parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 3 June 2020 at 21:01:48 UTC, Steven Schveighoffer 
wrote:

 But the question above is still unanswered -- writeln receives 
 a string as an immutable(char)[], and doesn't know that it's 
 from a literal.
Perhaps add an overload that takes `const(char)*`? Here's an example: string foo(T...)(T args) // signature of `writeln` { return "..."; } string foo(const(char)* value) { return "foo"; } void main() { assert(foo("bar") == "foo"); } The above assertion pass. That means the compiler prefers the version with `const(char)*` for string literals in favor over the template version. -- /Jacob Carlborg
Jun 04
parent Nick Treleaven <nick geany.org> writes:
On Thursday, 4 June 2020 at 11:13:12 UTC, Jacob Carlborg wrote:
 On Wednesday, 3 June 2020 at 21:01:48 UTC, Steven Schveighoffer 
 wrote:
 But the question above is still unanswered -- writeln receives 
 a string as an immutable(char)[], and doesn't know that it's 
 from a literal.
Perhaps add an overload that takes `const(char)*`?
That overload would be unsafe: const c = 'c'; writeln(&c); writeln(&"s".idup[0]);
Jun 04
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 19:59:31 UTC, Walter Bright wrote:
 On 6/3/2020 1:06 AM, Nick Treleaven wrote:
 How does writeln tell that a zero terminated string has been 
 passed?
D string literals always have a 0 terminator.
Indeed, but the type system cannot distinguish literals from other strings. That is, from writeln's perspective: writeln("hello"); and writeln("hello world"[0 .. 5]); are indistinguishable, yet the latter is not zero terminated. That's why writeln cannot just blindly forward to puts. It may be able to forward to fwrite() though... which is very close to what it actually does, it just checks some edge cases along the way which has a tiny cost in this case but buys us capabilities in other cases.
Jun 03
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 2:02 PM, Adam D. Ruppe wrote:
 That's why writeln cannot just blindly forward to puts. It may be able to 
 forward to fwrite() though... which is very close to what it actually does,
What it actually does is generate a blizzard of templates into the object code.
Jun 04
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:
 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.
iff it is a string literal and only a string literal. tho the zero-terminator can be fixed with fwrite of course.
 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.
It actually tries to special-case string literals (and it does a fine job - the compile time on my computer is scarcely different than using `core.stdc.stdio`), but turns out this is trickier than it might seem since there's a lot of edge cases in real generic D code. On Linux, this generic implementation costs about 0.04s in compile time. On Windows, it costs about 0.08s (which is probably more the cost of importing core.sys.windows.windows than anything else). tbh I think this case looks worse than it is.
Jun 03
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 14:04:23 UTC, Adam D. Ruppe wrote:
 On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:
 [...]
iff it is a string literal and only a string literal. tho the zero-terminator can be fixed with fwrite of course.
 [...]
It actually tries to special-case string literals (and it does a fine job - the compile time on my computer is scarcely different than using `core.stdc.stdio`), but turns out this is trickier than it might seem since there's a lot of edge cases in real generic D code. On Linux, this generic implementation costs about 0.04s in compile time. On Windows, it costs about 0.08s (which is probably more the cost of importing core.sys.windows.windows than anything else). tbh I think this case looks worse than it is.
Try to print an enum please.
Jun 03
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 14:10:21 UTC, Stefan Koch wrote:
 Try to print an enum please.
That's a separate case. There's branches of writeln that could definitely be improved. But the simple string one isn't too bad, and the 0.04s you pay for the check means things like printing enums actually work, even if that branch isn't as well done. puts has no hope of handling these cases.
Jun 03
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:
 tbh I think this case looks worse than it is.
Not if you look at the code generated by: import std.stdio; void test() { writeln("hello"); } It's embarrassing.
Jun 03
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 20:02:29 UTC, Walter Bright wrote:
 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:
 tbh I think this case looks worse than it is.
Not if you look at the code generated by: import std.stdio; void test() { writeln("hello"); } It's embarrassing.
Yes! and now look at std.traits and std.meta :)
Jun 03
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 20:38:48 UTC, Stefan Koch wrote:
 On Wednesday, 3 June 2020 at 20:02:29 UTC, Walter Bright wrote:
 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:
 tbh I think this case looks worse than it is.
Not if you look at the code generated by: import std.stdio; void test() { writeln("hello"); } It's embarrassing.
Yes! and now look at std.traits and std.meta :)
or at std.conv.to: See: https://www.youtube.com/watch?v=jNos&feature=youtu.be-Va01wA?t=1768
Jun 03
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/20 4:02 PM, Walter Bright wrote:
 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:
 tbh I think this case looks worse than it is.
Not if you look at the code generated by:   import std.stdio;   void test()   {     writeln("hello");   } It's embarrassing.
Here it is: https://godbolt.org/z/LjUtc4 I was surprised. It would be great if someone took a look at that.
Jun 05
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:
 On 6/3/20 4:02 PM, Walter Bright wrote:
 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:
 tbh I think this case looks worse than it is.
Not if you look at the code generated by:   import std.stdio;   void test()   {     writeln("hello");   } It's embarrassing.
Here it is: https://godbolt.org/z/LjUtc4 I was surprised. It would be great if someone took a look at that.
What surprised you?
Jun 05
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/5/20 4:12 PM, Stefan Koch wrote:
 On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:
 On 6/3/20 4:02 PM, Walter Bright wrote:
 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:
 tbh I think this case looks worse than it is.
Not if you look at the code generated by:    import std.stdio;    void test()    {      writeln("hello");    } It's embarrassing.
Here it is: https://godbolt.org/z/LjUtc4 I was surprised. It would be great if someone took a look at that.
What surprised you?
Code, code everywhere...
Jun 05
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 5 June 2020 at 20:18:01 UTC, Andrei Alexandrescu wrote:
 On 6/5/20 4:12 PM, Stefan Koch wrote:
 On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu 
 wrote:
 On 6/3/20 4:02 PM, Walter Bright wrote:
 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:
 tbh I think this case looks worse than it is.
Not if you look at the code generated by:    import std.stdio;    void test()    {      writeln("hello");    } It's embarrassing.
Here it is: https://godbolt.org/z/LjUtc4 I was surprised. It would be great if someone took a look at that.
What surprised you?
Code, code everywhere...
To me it looks like an accurate translation of the source code.
Jun 05
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:
 It would be great if someone took a look at that.
Yeah, this is why when reducing bugs I often prefer to use `printf` or even `assert(0)` or `asm { int 3; }` since it does result in a lot less code generated. BUT, in the context of a regular program, I caution against making too big a deal of this. Yes, there's more code than there needs to be *for this simple case*, but much of it exists to support real world code that is more complicated out of necessity. People don't actually write "hello world" programs. It'd be amusing to have the compiler recognize the input pattern and output some hand-tuned perfectly assembly for it.... but it'd also be useless. Once you start writing slices instead of literals, or enums and structs instead of basic types, much of that code is useful anyway. Not all of it, again, I know some of writeln's branches are horrible, and I personally think the wrapping through C's lib with additional thread safety on top of it is silly. But those decisions are all made for debatably good reasons and at the end of the day, is it worth worrying about 0.04s and 200 bytes on a hello world build?
Jun 05
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 05, 2020 at 08:41:53PM +0000, Adam D. Ruppe via Digitalmars-d wrote:
 On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:
 It would be great if someone took a look at that.
Yeah, this is why when reducing bugs I often prefer to use `printf` or even `assert(0)` or `asm { int 3; }` since it does result in a lot less code generated.
I guess YMMV and all that, but I find all the "unnecessary complications" of writefln eminently useful when debugging, esp. with the %(...%) format specs that can print ranges and can be nested. Very useful for one-line, throwaway debug dumps of range contents that doesn't require writing an entire function and range-interpretation framework just to get some miserable debug output, like you need to in C. With write(f)ln's "complications", you can just write a 1-line debug dump that you throw away after the next code-compile-debug cycle without thinking twice.
 BUT, in the context of a regular program, I caution against making too
 big a deal of this. Yes, there's more code than there needs to be *for
 this simple case*, but much of it exists to support real world code
 that is more complicated out of necessity.
Yes, we're blowing so much steam over a measly hello world program when larger, much more important issues lie unsolved. Typical bikeshed vs. nuclear reactor situation. :-D
 People don't actually write "hello world" programs. It'd be amusing to
 have the compiler recognize the input pattern and output some
 hand-tuned perfectly assembly for it.... but it'd also be useless.
[...] Even more amusing would be for the compiler to recognize the pattern, and launch a browser to some tutorial page on dlang.org explaining next steps to go after your hello world program runs. :-P But yeah, also equally useless. T -- The peace of mind---from knowing that viruses which exploit Microsoft system vulnerabilities cannot touch Linux---is priceless. -- Frustrated system administrator.
Jun 05
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 5 June 2020 at 21:07:26 UTC, H. S. Teoh wrote:
 I guess YMMV and all that, but I find all the "unnecessary 
 complications" of writefln eminently useful when debugging,
oh i agree, I like it when going through that process. writefln is p nice. just at the *end* of the process, when it comes down to reducing the bug to the smallest thing so we can inspect the disassembly and report it upstream, then I avoid it.
Jun 05
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 05, 2020 at 09:18:20PM +0000, Adam D. Ruppe via Digitalmars-d wrote:
 On Friday, 5 June 2020 at 21:07:26 UTC, H. S. Teoh wrote:
 I guess YMMV and all that, but I find all the "unnecessary
 complications" of writefln eminently useful when debugging,
oh i agree, I like it when going through that process. writefln is p nice. just at the *end* of the process, when it comes down to reducing the bug to the smallest thing so we can inspect the disassembly and report it upstream, then I avoid it.
Well, if you're after the disassembly, assert(0) seems to be the ticket. :-) T -- One reason that few people are aware there are programs running the internet is that they never crash in any significant way: the free software underlying the internet is reliable to the point of invisibility. -- Glyn Moody, from the article "Giving it all away"
Jun 05
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2020 1:41 PM, Adam D. Ruppe wrote:
 But those decisions are all made for debatably good reasons 
 and at the end of the day, is it worth worrying about 0.04s and 200 bytes on a 
 hello world build?
writeln() is a fundamental piece of Phobos, and should be done right. Back in the olden days, Borland's C compiler didn't produce very good code. But Borland managed to still get good results by doing the basics right - they went all out on hand-optimizing the C standard library. In particular, they implemented printf entirely in hand-optimized assembler. Since printf is used everywhere in C code, this paid of handsomely for Borland. AFAIK, they're the only C vendor that did this.
Jun 05
prev sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:

 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.

 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.
A good `writeln` would not call into C at all. Nor would it copy its arguments like four times (therefore potentially not printing what it was given). But Phobos' `writeln` is actually a `encumber_you_with_dependencies_and_thread_safety_because_it_must_be_good_for_you_ oundabout_writeln`. So it's OK, we're only missing a plain `writeln`.
Jun 03
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 3 June 2020 at 14:58:59 UTC, Stanislav Blinov wrote:
 On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:

 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.

 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.
A good `writeln` would not call into C at all. Nor would it copy its arguments like four times (therefore potentially not printing what it was given). But Phobos' `writeln` is actually a `encumber_you_with_dependencies_and_thread_safety_because_it_must_be_good_for_you_ oundabout_writeln`. So it's OK, we're only missing a plain `writeln`.
To be fair, C's `puts` also includes thread-safety. You need to use non-standard functions like `fputs_unlocked` if you want to avoid that overhead.
Jun 03
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 7:58 AM, Stanislav Blinov wrote:
 A good `writeln` would not call into C  at all.
C's puts() is thread safe. No reason to not use it.
 So it's OK, we're only missing a plain `writeln`.
Doing the basic, common cases by going around the horn is not good engineering.
Jun 03
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 20:05:39 UTC, Walter Bright wrote:
 On 6/3/2020 7:58 AM, Stanislav Blinov wrote:
 A good `writeln` would not call into C  at all.
C's puts() is thread safe. No reason to not use it.
A thread-safe writeln would be threadsafe_writeln. Or, alternatively there would be a writeln and a not_threadsafe_writeln. No reason to pay for something you're not using. And still neither implementation would call into C, the OS is right there. I guess this is where, in Phobos, one could use the output range format API with a custom sink. But that wouldn't help against all the unnecessary copying.
 So it's OK, we're only missing a plain `writeln`.
Doing the basic, common cases by going around the horn is not good engineering.
On that, we're in agreement.
Jun 03
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 4:29 PM, Stanislav Blinov wrote:
 On Wednesday, 3 June 2020 at 20:05:39 UTC, Walter Bright wrote:
 On 6/3/2020 7:58 AM, Stanislav Blinov wrote:
 A good `writeln` would not call into C  at all.
C's puts() is thread safe. No reason to not use it.
A thread-safe writeln would be threadsafe_writeln. Or, alternatively there would be a writeln and a not_threadsafe_writeln. No reason to pay for something you're not using. And still neither implementation would call into C, the OS is right there.
writeln writes to a FILE *, which is C. There's no way around it. FILE * is buffered i/o, so going around it would mean odd out-of-order output. -Steve
Jun 03
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 21:04:23 UTC, Steven Schveighoffer 
wrote:

 A thread-safe writeln would be threadsafe_writeln. Or, 
 alternatively there would be a writeln and a 
 not_threadsafe_writeln. No reason to pay for something you're 
 not using. And still neither implementation would call into C, 
 the OS is right there.
writeln writes to a FILE *, which is C. There's no way around it. FILE * is buffered i/o, so going around it would mean odd out-of-order output.
There's no way around writing to a FILE*? I... hope you didn't mean what you wrote, all things considered ;) I know what Phobos' writeln does. But whatever, this is not the topic. Maybe in another thread at another time...
Jun 03
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 5:29 PM, Stanislav Blinov wrote:
 On Wednesday, 3 June 2020 at 21:04:23 UTC, Steven Schveighoffer wrote:
 
 A thread-safe writeln would be threadsafe_writeln. Or, alternatively 
 there would be a writeln and a not_threadsafe_writeln. No reason to 
 pay for something you're not using. And still neither implementation 
 would call into C, the OS is right there.
writeln writes to a FILE *, which is C. There's no way around it. FILE * is buffered i/o, so going around it would mean odd out-of-order output.
There's no way around writing to a FILE*? I... hope you didn't mean what you wrote, all things considered ;)
Well, no way around it if you want writeln to do expected things ;) Unless you are suggesting that Phobos completely drop support for interleaving printf and writeln calls? If you are, I assure you Walter will not accept that.
 I know what Phobos' writeln does. But whatever, this is not the topic. 
 Maybe in another thread at another time...
Not sure why you think the implementation of Phobos using C for I/O is not on-topic. It was you who suggested we could avoid it for certain calls to writeln. -Steve
Jun 03
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 22:21:41 UTC, Steven Schveighoffer 
wrote:
 On 6/3/20 5:29 PM, Stanislav Blinov wrote:
 There's no way around writing to a FILE*? I... hope you didn't 
 mean what you wrote, all things considered ;)
Well, no way around it if you want writeln to do expected things ;) Unless you are suggesting that Phobos completely drop support for interleaving printf and writeln calls? If you are, I assure you Walter will not accept that.
Phobos can have whatever it has. That wasn't my point.
 Not sure why you think the implementation of Phobos using C for 
 I/O is not on-topic. It was you who suggested we could avoid it 
 for certain calls to writeln.
Because topic is template bloat, err... statistics :) And I'm well at fault for straying. I was speaking figuratively. A *good* writeln need not go to C - that was my retort to bringing up puts in 2020. Not *the* writeln, *a* writeln. At the very least it shouldn't be the default. My point is we should be striving for better, not that there's anything wrong with puts or C.
Jun 03
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 9:27 PM, Stanislav Blinov wrote:
 On Wednesday, 3 June 2020 at 22:21:41 UTC, Steven Schveighoffer wrote:
 On 6/3/20 5:29 PM, Stanislav Blinov wrote:
 There's no way around writing to a FILE*? I... hope you didn't mean 
 what you wrote, all things considered ;)
Well, no way around it if you want writeln to do expected things ;) Unless you are suggesting that Phobos completely drop support for interleaving printf and writeln calls? If you are, I assure you Walter will not accept that.
Phobos can have whatever it has. That wasn't my point.
Ah, I get it, sorry for misunderstanding of your point. Yes, I'm in the same boat. I've hated how Phobos uses FILE * for its I/O.
 
 Not sure why you think the implementation of Phobos using C for I/O is 
 not on-topic. It was you who suggested we could avoid it for certain 
 calls to writeln.
Because topic is template bloat, err... statistics :) And I'm well at fault for straying. I was speaking figuratively. A *good* writeln need not go to C - that was my retort to bringing up puts in 2020. Not *the* writeln, *a* writeln. At the very least it shouldn't be the default. My point is we should be striving for better, not that there's anything wrong with puts or C.
Agree on all that, sorry for the confusion. -Steve
Jun 03
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 3:21 PM, Steven Schveighoffer wrote:
 Unless you are suggesting that Phobos completely drop support for interleaving 
 printf and writeln calls? If you are, I assure you Walter will not accept that.
I just checked with him, and he assured me he won't. Also, a plain writeln("hello") must be thread safe just like puts("hello"). The reason is simply customer demand. I recall the bad old days when multiple threads would write to stdout, and you'd get: heworllldo and everyone would complain. People accept that printfs can be interleaved on the function call level, but not inside the call. Furthermore, stdout has to acquire a mutex before writing to it: acquire stdout mutex print 'h' release stdout mutex acquire stdout mutex print 'e' release stdout mutex acquire stdout mutex print 'l' release stdout mutex turns out to be disastrously slower than: acquire stdout mutex print 'hello' release stdout mutex
Jun 03
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 4 June 2020 at 01:38:50 UTC, Walter Bright wrote:
 Furthermore, stdout has to acquire a mutex before writing to it:
No need for any of this on the OS level, a write up to like 4k is guaranteed to be atomic. It is the C buffer that isn't :( And the C buffer is a consistent source of pain for user-interactive i/o. Buffering makes a big performance difference when you are actually outputting a block... but with user-interactive you rarely actually *want* buffering since users want to see their results ASAP! And remember, pipes are frequently user-interactive too and them not being flushed frequently is a FAQ among new users, especially IDE users. I kinda feel buffering should not be done across function boundaries for stdout and stderr. Other files maybe.
Jun 03
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 10:28 PM, Adam D. Ruppe wrote:
 On Thursday, 4 June 2020 at 01:38:50 UTC, Walter Bright wrote:
 Furthermore, stdout has to acquire a mutex before writing to it:
No need for any of this on the OS level, a write up to like 4k is guaranteed to be atomic. It is the C buffer that isn't :( And the C buffer is a consistent source of pain for user-interactive i/o. Buffering makes a big performance difference when you are actually outputting a block... but with user-interactive you rarely actually *want* buffering since users want to see their results ASAP! And remember, pipes are frequently user-interactive too and them not being flushed frequently is a FAQ among new users, especially IDE users. I kinda feel buffering should not be done across function boundaries for stdout and stderr. Other files maybe.
I don't think you are consider the ramifications. You NEED a buffer, otherwise, e.g. you will be issuing system calls for writing 5 bytes, followed by an immediate system call for 5 more. And that isn't cheap. Locking a mutex would be dwarfed by that. Flushing on newlines is a reasonable compromise. But it doesn't have to be a C buffer (unless you want to interact with printf). There are good reasons to use C buffering for the standard handles. But it sucks that ALL Phobos I/O has to be based on it (including opening a file that you never use in a C call). -Steve
Jun 03
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 4 June 2020 at 03:00:26 UTC, Steven Schveighoffer 
wrote:
 buffer, otherwise, e.g. you will be issuing system calls for 
 writing 5 bytes, followed by an immediate system call for 5 
 more.
Yeah, I know. You and I have both written low-level I/O code, we both know how it works and the huge difference proper buffering can make. But the question is where do you have implicit flushes? I'm arguing for at the end of the function. The call to writefln or whatever would do an intermediate buffer, but when it returns, it flushes it all at once. You don't flush intermediates if you can get away with it, but you treat each function as a direct write. If a user actually writes out `write("h"); write("e"); write("l");`, well, that's on them, maybe they wanted to see it spell out! But if they write `writefln("hello, %s, you have %d points!", name, score);`, the implementation should NOT do os.write("hello, "); os.write(name); os.write(", you have "); no, it should go ahead and buffer those pieces, then at the very end write it all. Just again, if the USER writes it out in separate calls... well, that's on them. This will match what happens in most cases anyway since the C buffer DOES flush on \n most the time (and when it doesn't, users complain and I have to explain it to them AGAIN) getting a lil off topic for templates btw.......
Jun 03
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 11:14 PM, Adam D. Ruppe wrote:

 getting a lil off topic for templates btw.......
Yeah, for sure. My one final point would be that sometimes writing things out piecemeal based on logic is natural. I don't think the user should have to do acrobatics to fit it all in one call. BTW, I misunderstood your original point, and thought you meant stdout and stderr should not be buffered at all. -Steve
Jun 03
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 7:28 PM, Adam D. Ruppe wrote:
 I kinda feel buffering should not be done across function boundaries for
stdout 
 and stderr. Other files maybe.
That's why there's "isatty()".
Jun 03
prev sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Thursday, 4 June 2020 at 01:38:50 UTC, Walter Bright wrote:
 On 6/3/2020 3:21 PM, Steven Schveighoffer wrote:
 Unless you are suggesting that Phobos completely drop support 
 for interleaving printf and writeln calls? If you are, I 
 assure you Walter will not accept that.
I just checked with him, and he assured me he won't.
:)
 Also, a plain writeln("hello") must be thread safe just like 
 puts("hello"). The reason is simply customer demand. I recall 
 the bad old days when multiple threads would write to stdout, 
 and you'd get:

   heworllldo
That's simple, multiple threads shouldn't be writing to stdout ;) And that's only a half-joke. There of course must be a thread-safe writeln, just as there must be a non-thread-safe writeln (with different names, obviously). Which is the default should depend on the program. If you're only running one thread, or more to the point, only doing stdout from one thread [at a time], you have no use for those kinds of locks. It's just waste of cache. If you're not using any C libraries, or ones you are using aren't doing stdout, you have no use for FILE*. The OS is right there, a syscall away. Phobos does make provisions for rolling your own sink (formattedWrite), which is great (more templates though :P). But, when you do sit down to write your own, core.sys is startlingly deprived of sys and is full of C. And now it becomes a roll your own syscalls... And then you find core.osthread full of pthread and not osthread.
Jun 03
prev sibling parent Max Samukha <maxsamukha gmail.com> writes:
On Wednesday, 3 June 2020 at 20:05:39 UTC, Walter Bright wrote:
 On 6/3/2020 7:58 AM, Stanislav Blinov wrote:
 A good `writeln` would not call into C  at all.
C's puts() is thread safe. No reason to not use it.
 So it's OK, we're only missing a plain `writeln`.
Doing the basic, common cases by going around the horn is not good engineering.
Superfluous special cases and premature optimization is not good software engineering either.
Jun 03
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 3 June 2020 at 14:58:59 UTC, Stanislav Blinov wrote:

 A good `writeln` would not call into C  at all.
What would it do then? Call syscalls in the kernel directly? That's more or less only supported on Linux. It's definitely not supported on macOS. You can do it, but it's not supported and make break at any time. Go used to do that, something broke for them and now they're going through the C wrappers. -- /Jacob Carlborg
Jun 04
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Thursday, 4 June 2020 at 09:10:06 UTC, Jacob Carlborg wrote:
 On Wednesday, 3 June 2020 at 14:58:59 UTC, Stanislav Blinov 
 wrote:

 A good `writeln` would not call into C  at all.
What would it do then? Call syscalls in the kernel directly? That's more or less only supported on Linux. It's definitely not supported on macOS. You can do it, but it's not supported and make break at any time. Go used to do that, something broke for them and now they're going through the C wrappers.
Going through the system's library to *make syscalls*, yes. Obviously I'm presenting an idealized case. If you can't sustain direct interop with the OS to maintain compatibility, then yes, while unfortunate, you'd use what you have to. But that is a far cry from unconditionally routing your stdout through FILE*. The situation with Mac only highlights how infectuous C actually is. And Linux is only slightly better, to be honest.
Jun 05
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 3 June 2020 at 02:32:54 UTC, Walter Bright wrote:
 On 6/2/2020 5:53 AM, Ethan wrote:
 Oh I have *got* to try this out on my code.
I tried it on `writeln("hello");` and the number of templates instantiated seems excessive. It should be investigated. Number Unique Name 46 38 MAKEINTRESOURCE_T(ushort i) 38 0 OldAlias(T) if (!isAggregateType!T || is(Unqual!T == T)) 7 6 staticIndexOf(T, TList...) 1 1 Impl(T) 7 5 isSomeChar(T) 2 1 StringTypeOf(T) 1 1 Impl(T) 8 2 mostNegative(T) if (isNumeric!T || isSomeChar!T || isBoolean!T) 1 1 encode(Flag useReplacementDchar = No.useReplacementDchar)(out char[4] buf, dchar c) 7 7 ModifyTypePreservingTQ(alias Modifier, T) 2 1 Unqual(T) 1 1 put(A)(A writeme) if ((isSomeChar!(Unqual!(ElementType!A)) || is(ElementType!A : const(ubyte))) && i sInputRange!A && !isInfinite!A) 1 1 decodeImpl(bool canIndex, Flag useReplacementDchar = No.useReplacementDchar, S)(auto ref S str, ref size_t index) if (is(S : const(char[])) || isInputRange!S && is(Unqual!(ElementEncodingType!S) == char)) 3 1 isCallable(T...) if (T.length == 1) 1 1 trustedFwrite(T)(FILE* f, const T[] obj) 2 1 _utfException(Flag useReplacementDchar)(string msg, dchar c) 1 1 popFront(C)(ref C[] str) if (Autodecoding && isNarrowString!(C[])) 2 2 IntegralTypeOf(T) 1 1 Impl(T) 1 1 Impl(T) 20 20 genericIndexOf(args...) if (args.length >= 1) 1 1 safeOp(T0, T1)(auto ref T0 a, auto ref T1 b) 5 1 ElementEncodingType(R) 1 1 errnoEnforce(T, string file = __FILE__, uint line = __LINE__)(T value, lazy string msg = null) 1 1 Impl(T) 2 2 put(C)(C c) if (isSomeChar!C || is(C : const(ubyte))) 2 1 exception(S)(S str, string msg) 6 1 isNarrowString(T) 2 1 at(R)(R[] r, size_t i) 1 1 FunctionTypeOf(func...) if (func.length == 1 && isCallable!func) 1 1 MinType(T...) if (T.length >= 1) 5 0 AliasSeq(TList...) 4 1 opDispatch(string name) 1 1 min(T...)(T args) if (T.length >= 2) 8 8 sndAlias(char c0, char c1) 1 1 __equals(T1, T2)(T1[] lhs, T2[] rhs) 55 13 isAggregateType(T) 8 2 isNumeric(T) 7 7 OriginalType(T) 4 1 isInputRange(R) 1 1 empty(T)(in T[] a) 6 3 ElementType(R) 36 11 expectType(T) 24 7 Unqual(T) 1 1 MAKEINTATOM_T()(int i) 1 1 writeln(T...)(T args) 1 1 Impl(T) 3 1 isSomeString(T) 1 1 isSomeFunction(T...) if (T.length == 1) 15 2 TypeDef(T) 1 1 isInfinite(R) 3 1 isStaticArray(T) 18 18 isSame(ab...) if (ab.length == 2) 2 1 front(T)(T[] a) if (Autodecoding && isNarrowString!(T[])) 1 1 decode(Flag useReplacementDchar = No.useReplacementDchar, S)(auto ref S str, ref size_t index) if (i sSomeString!S) 1 1 codeUnitLimit(S) if (isSomeChar!(ElementEncodingType!S)) 2 1 Parameters(func...) if (func.length == 1 && isCallable!func) 1 1 fd_set_custom(uint SETSIZE) 4 4 mmioFOURCC(char c0, char c1, char c2, char c3) 2 1 Demangle(T) 4 3 Flag(string name) 1 1 safeOp(string S) if (S == "<" || S == ">" || S == "<=" || S == ">=" || S == "==" || S == "!=") 2 2 isIntegral(T) 1 1 Impl(T) 5 5 CharTypeOf(T)
looks like abstractions are not zero cost in fine
Jun 02
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 03:06:37 UTC, Basile B. wrote:
 On Wednesday, 3 June 2020 at 02:32:54 UTC, Walter Bright wrote:
        [...]
looks like abstractions are not zero cost in fine
Looks like you haven't used `-vcg-ast` either. If you do you'll see what's going on. Some of these can be replaced with CTFE. Generally only things that need to work on template parameter lists are forced to be templates. Well and legitimate type parameterization of course. But those are the minority. If the template sounds like a question about a type, Like (CharTypeof or isIntegral) it probably should not be a template.
Jun 02
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 3 June 2020 at 04:52:37 UTC, Stefan Koch wrote:
 [snip]

 If the template sounds like a question about a type,
 Like (CharTypeof or isIntegral) it probably should not be a 
 template.
I was a little confused by this. Given your recent work on type functions, do you mean that type functions should replace templates in situations like that? That would imply that things like `isIntegral` has to be a template now, but could be changed in the future, correct? Do you think something like mir's `sumType` [1] should not be a template? [1] https://github.com/libmir/mir-algorithm/blob/f111c311eadb0057f5a233c156b772a010ceab71/source/mir/math/sum.d#L1986
Jun 03
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 11:09:56 UTC, jmh530 wrote:
 On Wednesday, 3 June 2020 at 04:52:37 UTC, Stefan Koch wrote:
 [snip]

 If the template sounds like a question about a type,
 Like (CharTypeof or isIntegral) it probably should not be a 
 template.
I was a little confused by this. Given your recent work on type functions, do you mean that type functions should replace templates in situations like that? That would imply that things like `isIntegral` has to be a template now, but could be changed in the future, correct? Do you think something like mir's `sumType` [1] should not be a template? [1] https://github.com/libmir/mir-algorithm/blob/f111c311eadb0057f5a233c156b772a010ceab71/source/mir/math/sum.d#L1986
Oh yes. sumType does not need to introduce new symbols. Therefore it should be handled in a way which does not introduce new symbols. type functions are one way to make this happen. However changes to the language are hard justify therefore I am currently revisiting, approaches to infer type-function-like behavior and transform them behind the scenes. (Which is a HUGE pain, because you have to transform recursion into loops, which is a hard problem in itself, AND the user probably would rather have written a loop, but was forced by the language to use recursion. ) alas having my work being useful in the language is more important than to have a better language. I will continue on type functions; if the conservative approaches don't work out.
Jun 03
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:
 Oh yes.
 sumType does not need to introduce new symbols.
 Therefore it should be handled in a way which does not 
 introduce new symbols.
In other words, we want a way to give our templates AST-macro-like evaluation semantics. It's the same thing with Manu's `...` and staticMap. :) I wonder if a DIP to add something like "Macros: the Good Parts" to D would be accepted?
Jun 03
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 14:13:47 UTC, Paul Backus wrote:
 On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:
 Oh yes.
 sumType does not need to introduce new symbols.
 Therefore it should be handled in a way which does not 
 introduce new symbols.
In other words, we want a way to give our templates AST-macro-like evaluation semantics. It's the same thing with Manu's `...` and staticMap. :) I wonder if a DIP to add something like "Macros: the Good Parts" to D would be accepted?
I doubt that it would have any more success than type functions.
Jun 03
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:
[snip]
 Oh yes.
 sumType does not need to introduce new symbols.
 Therefore it should be handled in a way which does not 
 introduce new symbols.

 type functions are one way to make this happen.
 However changes to the language are hard justify therefore I am 
 currently revisiting,
 approaches to infer type-function-like behavior and transform 
 them behind the scenes.
 (Which is a HUGE pain, because you have to transform recursion 
 into loops, which is a hard problem in itself, AND the user 
 probably would rather have written a loop, but was forced by 
 the language to use recursion. )

 alas having my work being useful in the language is more 
 important than to have a better language.

 I will continue on type functions; if the conservative 
 approaches don't work out.
Thanks. The problem almost sounds like you need an inliner for types in the compiler.
Jun 03
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:
 AND the user probably would rather have written a loop, but was 
 forced by the language to use recursion.
That's definitely true.
 I will continue on type functions; if the conservative 
 approaches don't work out.
I hope we can get type functions. -- /Jacob Carlborg
Jun 04
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 4 June 2020 at 08:38:59 UTC, Jacob Carlborg wrote:
 On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:
 AND the user probably would rather have written a loop, but 
 was forced by the language to use recursion.
That's definitely true.
 I will continue on type functions; if the conservative 
 approaches don't work out.
I hope we can get type functions. -- /Jacob Carlborg
Well to be completely honest type functions do look cool. But they to have a their own set of problems. I designed them so it would be impossible to "abuse" them. I did that by putting a restriction in that would disallow them from generating symbols. And that does work and greatly simplifies the implementation. However, it also so means that they can't be used in many places that you would like to use them. For example creating a std.typecons.Tuple is impossible with type functions alone. For such a task they can only be used to reduce the intermediate symbols within a template. Which I fear will lead to some frustration. Also you would need to rewrite your entire reflection framework to use typefunctions. For that to be feasible new __traits have to be introduced. ..... All in all even though the feature alone is not that heavy, it will need major additions to be useful. I cannot do this alone. The way I see it; If Walter is not in the boat the boat will not get anywhere. It's to heavy for me to move by myself.
Jun 04
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:
 [snip]
 type functions are one way to make this happen.
 However changes to the language are hard justify therefore I am 
 currently revisiting,
 approaches to infer type-function-like behavior and transform 
 them behind the scenes.
 (Which is a HUGE pain, because you have to transform recursion 
 into loops, which is a hard problem in itself, AND the user 
 probably would rather have written a loop, but was forced by 
 the language to use recursion. )
 [snip]
I was just thinking about this again in light of Manu's recent post [1] on the bonus thread. Manu specifically mentions types not working well in static contexts like types and aliases. It occurs to me that there may be a connection between what Manu said and the issue that DIP 1023 [2] attempted to resolve. For instance, consider the DIP example alias PackedUpperTriangularMatrix(T) = Slice!(StairsIterator!(T*, "-")); What if instead you have a function that takes a type and returns an alias to another type? Ignoring any issues with the syntax, it might look something like this alias PackedUpperTriangularMatrix(T)() { return alias Slice!(StairsIterator!(T*, "-")); } and could be used like auto foo(T)(PackedUpperTriangularMatrix!T m) { } (which is the same as in the DIP). At the end of the day, you would still need to ensure the compiler would actually resolve those types before the function is called. So I imagine that just adding the support to be able to do the first bit does not imply the second bit would work. However, the issue with the solution presented in DIP 1023 was that it was only designed to work with the short alias syntax and not the full template alias syntax, which may not have been possible and could have led to a special casing. This is a simple rule: resolve the type/alias functions first. I know on other threads you have talked about how much work it would be to go further with type functions and you would need Walter's blessing and help. However, I think that if it means that we could get an alternative resolution to the issues brought up by DIP 1023, then I would be all on board (and willing to contribute to support the work). [1] https://forum.dlang.org/post/mailman.3585.1591274099.31109.digitalmars-d puremagic.com [2] https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1023.md
Jun 04
prev sibling next sibling parent Atwork <fwafwa fwfafa.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
This makes me happy, thank you!
Jun 03
prev sibling next sibling parent James Lu <jamtlu gmail.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
What if DMD had an opt-in option to gather and send detailed profiling data on compiles automatically? (If someone wanted to they could use Differential Privacy to randomize the numbers while preserving the overall informational content.)
Jun 05
prev sibling parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Looks super useful. I tried it on the test suite of openmethods and I see lots of duplicate entries, e.g.: [...] 2 1 setName(NewValue...) 5 5 UnpackDeepImpl(U...) 121 40 Parameters(func...) if (func.length == 1 && isCallable!func) 8 8 UnpackDeepImpl(U...) 2 2 argumentMixtureAt(int i) 3 3 UnpackDeepImpl(U...) 4 4 UnpackDeepImpl(U...) 5 5 UnpackDeepImpl(U...) [...]
Jun 06
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 6 June 2020 at 12:54:11 UTC, Jean-Louis Leroy wrote:
 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Looks super useful. I tried it on the test suite of openmethods and I see lots of duplicate entries, e.g.: [...] 2 1 setName(NewValue...) 5 5 UnpackDeepImpl(U...) 121 40 Parameters(func...) if (func.length == 1 && isCallable!func) 8 8 UnpackDeepImpl(U...) 2 2 argumentMixtureAt(int i) 3 3 UnpackDeepImpl(U...) 4 4 UnpackDeepImpl(U...) 5 5 UnpackDeepImpl(U...) [...]
Duplicate entries mean different Template Declarations with the same name. Usually those happen when templates are nested. Consider: template X(uint I1) { template Y() { alias Y = I1; } alias X = Y!(); } In this case the inner declaration is not dependent on any parameter that was passed to it. But it is dependent on the context in which it is instantiated. Every Instance of X will create a new separate context, In which a unique declaration of Y is formed.
Jun 06
parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Saturday, 6 June 2020 at 13:23:38 UTC, Stefan Koch wrote:
 On Saturday, 6 June 2020 at 12:54:11 UTC, Jean-Louis Leroy 
 wrote:
 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:
 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208
Looks super useful. I tried it on the test suite of openmethods and I see lots of duplicate entries, e.g.: [...] 2 1 setName(NewValue...) 5 5 UnpackDeepImpl(U...) 121 40 Parameters(func...) if (func.length == 1 && isCallable!func) 8 8 UnpackDeepImpl(U...) 2 2 argumentMixtureAt(int i) 3 3 UnpackDeepImpl(U...) 4 4 UnpackDeepImpl(U...) 5 5 UnpackDeepImpl(U...) [...]
Duplicate entries mean different Template Declarations with the same name. Usually those happen when templates are nested. Consider: template X(uint I1) { template Y() { alias Y = I1; } alias X = Y!(); } In this case the inner declaration is not dependent on any parameter that was passed to it. But it is dependent on the context in which it is instantiated. Every Instance of X will create a new separate context, In which a unique declaration of Y is formed.
I suspected something like this. I am trying to aggregate the template stats across several translation units (in fact the test suite of openmethods), to measure whether an implementation is better or worse than another of the same thing. In your example, what happens if I make a copy of X, name it Z, but leave the body unchanged? I won't be able to distinguish the X.Ys from the Z.Ys. Even within the same template, imagine I put different definitions of Y inside a `static if else`. I could not tell how many times each was used. So maybe template stats should display the fully qualified name (including the module) along with the line and column numbers. Maybe I'm missing something, I will try to find time to experiment more during the week-end.
Jun 06
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 6 June 2020 at 13:59:38 UTC, Jean-Louis Leroy wrote:
 On Saturday, 6 June 2020 at 13:23:38 UTC, Stefan Koch wrote:
 [...]
I suspected something like this. I am trying to aggregate the template stats across several translation units (in fact the test suite of openmethods), to measure whether an implementation is better or worse than another of the same thing. In your example, what happens if I make a copy of X, name it Z, but leave the body unchanged? I won't be able to distinguish the X.Ys from the Z.Ys. Even within the same template, imagine I put different definitions of Y inside a `static if else`. I could not tell how many times each was used. So maybe template stats should display the fully qualified name (including the module) along with the line and column numbers. Maybe I'm missing something, I will try to find time to experiment more during the week-end.
Adding the location is a good idea. I am going to open a PR. Fully qualified names however can absolutely gigantic for pathological cases, and I'd like to avoid that.
Jun 06