www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - template statistics

reply Walter Bright <newshound2 digitalmars.com> writes:
Many people are trying to figure out what templates are blowing up their 
compile. This should help:

https://github.com/dlang/dmd/pull/11208
Jun 01 2020
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


Nice!

Wishlist: if you could dump out a template instantiation 
backtrace combined with system time for every template 
declaration start and end, it should be easy to convert this into 
callgrind format ( 
https://valgrind.org/docs/manual/cl-format.html ), which would 
let us use graphical profilers with it.
Jun 02 2020
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:

 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


 Nice!

 Wishlist: if you could dump out a template instantiation 
 backtrace combined with system time for every template 
 declaration start and end, it should be easy to convert this 
 into callgrind format ( 
 https://valgrind.org/docs/manual/cl-format.html ), which would 
 let us use graphical profilers with it.


Ah I didn't know the callgrind format was documented :)
I can give you that I already have something similar for my own 
profiling.
Jun 02 2020
prev sibling next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:

 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


 Nice!

 Wishlist: if you could dump out a template instantiation 
 backtrace combined with system time for every template 
 declaration start and end, it should be easy to convert this 
 into callgrind format ( 
 https://valgrind.org/docs/manual/cl-format.html ), which would 
 let us use graphical profilers with it.


Check this branch out.
https://github.com/UplinkCoder/dmd/tree/dmd_tracing_2092
It doesn't do the cl-format.
But it should help in showing how-to add such per-symbol 
performance info
Jun 02 2020
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:

 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


 Nice!

 Wishlist: if you could dump out a template instantiation 
 backtrace combined with system time for every template 
 declaration start and end, it should be easy to convert this 
 into callgrind format ( 
 https://valgrind.org/docs/manual/cl-format.html ), which would 
 let us use graphical profilers with it.


Update: I've hacked up an extremely basic callgrind dump, just to 
test the format. It's based on line numbers, not functions, which 
callgrind doesn't like: "functions" are 
"file_filename_line_linenr". 
https://github.com/FeepingCreature/dmd/commit/aa379ba7750b8d32f359
6dc93d3c381cfa90e45 Note I'm *really* not sure if I'm doing the
inclusive/exclusive accounting correctly.

Did some internal testing. format!() seems to eat up a lot 
(40%?), not sure if I'm reading this right, but it wouldn't 
surprise me.
Jun 03 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 08:40:18 UTC, FeepingCreature wrote:

 On Tuesday, 2 June 2020 at 08:37:35 UTC, FeepingCreature wrote:

 [...]


 Update: I've hacked up an extremely basic callgrind dump, just 
 to test the format. It's based on line numbers, not functions, 
 which callgrind doesn't like: "functions" are 
 "file_filename_line_linenr". 
 https://github.com/FeepingCreature/dmd/commit/aa379ba7750b8d32f359
6dc93d3c381cfa90e45 Note I'm *really* not sure if I'm doing the
inclusive/exclusive accounting correctly.

 Did some internal testing. format!() seems to eat up a lot 
 (40%?), not sure if I'm reading this right, but it wouldn't 
 surprise me.


Looks correct.
Format is awful.
Jun 03 2020
prev sibling next sibling parent reply Ethan <gooberman gmail.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


Oh I have *got* to try this out on my code.

(Just need to find the time right now...)
Jun 02 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/2/2020 5:53 AM, Ethan wrote:

 Oh I have *got* to try this out on my code.


I tried it on `writeln("hello");` and the number of templates instantiated
seems 
excessive. It should be investigated.

   Number   Unique   Name
       46       38   MAKEINTRESOURCE_T(ushort i)
       38        0   OldAlias(T) if (!isAggregateType!T || is(Unqual!T == T))
        7        6   staticIndexOf(T, TList...)
        1        1   Impl(T)
        7        5   isSomeChar(T)
        2        1   StringTypeOf(T)
        1        1   Impl(T)
        8        2   mostNegative(T) if (isNumeric!T || isSomeChar!T ||
isBoolean!T)
        1        1   encode(Flag useReplacementDchar = 
No.useReplacementDchar)(out char[4] buf, dchar c)
        7        7   ModifyTypePreservingTQ(alias Modifier, T)
        2        1   Unqual(T)
        1        1   put(A)(A writeme) if ((isSomeChar!(Unqual!(ElementType!A)) 
|| is(ElementType!A : const(ubyte))) && i
sInputRange!A && !isInfinite!A)
        1        1   decodeImpl(bool canIndex, Flag useReplacementDchar = 
No.useReplacementDchar, S)(auto ref S str, ref
size_t index) if (is(S : const(char[])) || isInputRange!S && 
is(Unqual!(ElementEncodingType!S) == char))
        3        1   isCallable(T...) if (T.length == 1)
        1        1   trustedFwrite(T)(FILE* f, const T[] obj)
        2        1   _utfException(Flag useReplacementDchar)(string msg, dchar
c)
        1        1   popFront(C)(ref C[] str) if (Autodecoding && 
isNarrowString!(C[]))
        2        2   IntegralTypeOf(T)
        1        1   Impl(T)
        1        1   Impl(T)
       20       20   genericIndexOf(args...) if (args.length >= 1)
        1        1   safeOp(T0, T1)(auto ref T0 a, auto ref T1 b)
        5        1   ElementEncodingType(R)
        1        1   errnoEnforce(T, string file = __FILE__, uint line = 
__LINE__)(T value, lazy string msg = null)
        1        1   Impl(T)
        2        2   put(C)(C c) if (isSomeChar!C || is(C : const(ubyte)))
        2        1   exception(S)(S str, string msg)
        6        1   isNarrowString(T)
        2        1   at(R)(R[] r, size_t i)
        1        1   FunctionTypeOf(func...) if (func.length == 1 && 
isCallable!func)
        1        1   MinType(T...) if (T.length >= 1)
        5        0   AliasSeq(TList...)
        4        1   opDispatch(string name)
        1        1   min(T...)(T args) if (T.length >= 2)
        8        8   sndAlias(char c0, char c1)
        1        1   __equals(T1, T2)(T1[] lhs, T2[] rhs)
       55       13   isAggregateType(T)
        8        2   isNumeric(T)
        7        7   OriginalType(T)
        4        1   isInputRange(R)
        1        1   empty(T)(in T[] a)
        6        3   ElementType(R)
       36       11   expectType(T)
       24        7   Unqual(T)
        1        1   MAKEINTATOM_T()(int i)
        1        1   writeln(T...)(T args)
        1        1   Impl(T)
        3        1   isSomeString(T)
        1        1   isSomeFunction(T...) if (T.length == 1)
       15        2   TypeDef(T)
        1        1   isInfinite(R)
        3        1   isStaticArray(T)
       18       18   isSame(ab...) if (ab.length == 2)
        2        1   front(T)(T[] a) if (Autodecoding && isNarrowString!(T[]))
        1        1   decode(Flag useReplacementDchar = No.useReplacementDchar, 
S)(auto ref S str, ref size_t index) if (i
sSomeString!S)
        1        1   codeUnitLimit(S) if (isSomeChar!(ElementEncodingType!S))
        2        1   Parameters(func...) if (func.length == 1 &&
isCallable!func)
        1        1   fd_set_custom(uint SETSIZE)
        4        4   mmioFOURCC(char c0, char c1, char c2, char c3)
        2        1   Demangle(T)
        4        3   Flag(string name)
        1        1   safeOp(string S) if (S == "<" || S == ">" || S == "<=" ||
S 
== ">=" || S == "==" || S == "!=")
        2        2   isIntegral(T)
        1        1   Impl(T)
        5        5   CharTypeOf(T)
Jun 02 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 02:32:54 UTC, Walter Bright wrote:

       46       38   MAKEINTRESOURCE_T(ushort i)


This is a ported C macro in the Windows headers; a fixed cost (as 
in it won't grow with your program).

But it also doesn't have to exist - it could just be a CTFE 
function, or even just written out inline (all it does is cast a 
value to a different type!)



       38        0   OldAlias(T) if (!isAggregateType!T || 
 is(Unqual!T == T))


Now this one is a private template inside std.traits and I 
honestly don't understand why it exists. I suspect it was to 
paper over a compiler bug in the past and probably is now just 
unchanged out of fear of breakage.


Most the rest are reflection helpers in Phobos, many of which 
could prolly be trivially replaced with inline checks but .... 
maybe not a big deal anyway. idk.
Jun 02 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/2/2020 7:41 PM, Adam D. Ruppe wrote:

 Most the rest are reflection helpers in Phobos, many of which could prolly be 
 trivially replaced with inline checks but .... maybe not a big deal anyway.
idk.


The thing is, `writeln("hello")` can be expanded to 
`core.stdc.stdio.puts("hello")`. Done.

Phobos seems to do a lot of "going around the Horn" instead of taking the canal.
Jun 02 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:

 On 6/2/2020 7:41 PM, Adam D. Ruppe wrote:

 Most the rest are reflection helpers in Phobos, many of which 
 could prolly be trivially replaced with inline checks but .... 
 maybe not a big deal anyway. idk.


 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.

 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.


I have been saying that for years.

Apparently you never used `-vcg-ast`.
Jun 02 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 04:46:54 UTC, Stefan Koch wrote:

 On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:

 On 6/2/2020 7:41 PM, Adam D. Ruppe wrote:

 Most the rest are reflection helpers in Phobos, many of which 
 could prolly be trivially replaced with inline checks but 
 .... maybe not a big deal anyway. idk.


 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.

 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.


 I have been saying that for years.

 Apparently you never used `-vcg-ast`.


I've recently been informed that this may have sounded dismissive.
It was not meant that way.

I am just a little frustrated that my work to make this issue 
visible seems to have gone unnoticed.

Then again .... I didn't exactly advertise it.
Jun 04 2020
prev sibling next sibling parent reply Nick Treleaven <nick geany.org> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:

 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.


How does writeln tell that a zero terminated string has been 
passed?
Jun 03 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 1:06 AM, Nick Treleaven wrote:

 How does writeln tell that a zero terminated string has been passed?


D string literals always have a 0 terminator.
Jun 03 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 3:59 PM, Walter Bright wrote:

 On 6/3/2020 1:06 AM, Nick Treleaven wrote:

 How does writeln tell that a zero terminated string has been passed?


 
 D string literals always have a 0 terminator.


All compile-time strings should be both 0 terminated and implicitly 
convertible to immutable(char)*. I think there are some cases that 
aren't, but I don't remember all of them.

But the question above is still unanswered -- writeln receives a string 
as an immutable(char)[], and doesn't know that it's from a literal.

For example, if writeln blindly assumed that all immutable(char)[] were 
string literals, that it could pass to puts, this would be a problem:

writeln(somestr.idup);

-Steve
Jun 03 2020
parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 3 June 2020 at 21:01:48 UTC, Steven Schveighoffer 
wrote:


 But the question above is still unanswered -- writeln receives 
 a string as an immutable(char)[], and doesn't know that it's 
 from a literal.


Perhaps add an overload that takes `const(char)*`? Here's an 
example:

string foo(T...)(T args) // signature of `writeln`
{
     return "...";
}

string foo(const(char)* value)
{
     return "foo";
}

void main()
{
     assert(foo("bar") == "foo");
}

The above assertion pass. That means the compiler prefers the 
version with `const(char)*` for string literals in favor over the 
template version.

--
/Jacob Carlborg
Jun 04 2020
parent Nick Treleaven <nick geany.org> writes:
On Thursday, 4 June 2020 at 11:13:12 UTC, Jacob Carlborg wrote:

 On Wednesday, 3 June 2020 at 21:01:48 UTC, Steven Schveighoffer 
 wrote:

 But the question above is still unanswered -- writeln receives 
 a string as an immutable(char)[], and doesn't know that it's 
 from a literal.


 Perhaps add an overload that takes `const(char)*`?


That overload would be unsafe:

const c = 'c';
writeln(&c);
writeln(&"s".idup[0]);
Jun 04 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 19:59:31 UTC, Walter Bright wrote:

 On 6/3/2020 1:06 AM, Nick Treleaven wrote:

 How does writeln tell that a zero terminated string has been 
 passed?


 D string literals always have a 0 terminator.


Indeed, but the type system cannot distinguish literals from 
other strings.

That is, from writeln's perspective:

writeln("hello");

and

writeln("hello world"[0 .. 5]);

are indistinguishable, yet the latter is not zero terminated.

That's why writeln cannot just blindly forward to puts. It may be 
able to forward to fwrite() though... which is very close to what 
it actually does, it just checks some edge cases along the way 
which has a tiny cost in this case but buys us capabilities in 
other cases.
Jun 03 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 2:02 PM, Adam D. Ruppe wrote:

 That's why writeln cannot just blindly forward to puts. It may be able to 
 forward to fwrite() though... which is very close to what it actually does,


What it actually does is generate a blizzard of templates into the object code.
Jun 04 2020
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:

 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.


iff it is a string literal and only a string literal. tho the 
zero-terminator can be fixed with fwrite of course.


 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.


It actually tries to special-case string literals (and it does a 
fine job - the compile time on my computer is scarcely different 
than using `core.stdc.stdio`), but turns out this is trickier 
than it might seem since there's a lot of edge cases in real 
generic D code.

On Linux, this generic implementation costs about 0.04s in 
compile time. On Windows, it costs about 0.08s (which is probably 
more the cost of importing core.sys.windows.windows than anything 
else).

tbh I think this case looks worse than it is.
Jun 03 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 14:04:23 UTC, Adam D. Ruppe wrote:

 On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:

 [...]


 iff it is a string literal and only a string literal. tho the 
 zero-terminator can be fixed with fwrite of course.


 [...]


 It actually tries to special-case string literals (and it does 
 a fine job - the compile time on my computer is scarcely 
 different than using `core.stdc.stdio`), but turns out this is 
 trickier than it might seem since there's a lot of edge cases 
 in real generic D code.

 On Linux, this generic implementation costs about 0.04s in 
 compile time. On Windows, it costs about 0.08s (which is 
 probably more the cost of importing core.sys.windows.windows 
 than anything else).

 tbh I think this case looks worse than it is.


Try to print an enum please.
Jun 03 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 3 June 2020 at 14:10:21 UTC, Stefan Koch wrote:

 Try to print an enum please.


That's a separate case. There's branches of writeln that could 
definitely be improved.

But the simple string one isn't too bad, and the 0.04s you pay 
for the check means things like printing enums actually work, 
even if that branch isn't as well done.

puts has no hope of handling these cases.
Jun 03 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:

 tbh I think this case looks worse than it is.


Not if you look at the code generated by:

   import std.stdio;

   void test()
   {
     writeln("hello");
   }

It's embarrassing.
Jun 03 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 20:02:29 UTC, Walter Bright wrote:

 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:

 tbh I think this case looks worse than it is.


 Not if you look at the code generated by:

   import std.stdio;

   void test()
   {
     writeln("hello");
   }

 It's embarrassing.


Yes!

and now look at std.traits and std.meta :)
Jun 03 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 20:38:48 UTC, Stefan Koch wrote:

 On Wednesday, 3 June 2020 at 20:02:29 UTC, Walter Bright wrote:

 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:

 tbh I think this case looks worse than it is.


 Not if you look at the code generated by:

   import std.stdio;

   void test()
   {
     writeln("hello");
   }

 It's embarrassing.


 Yes!

 and now look at std.traits and std.meta :)


or at std.conv.to:

See: 
https://www.youtube.com/watch?v=jNos&feature=youtu.be-Va01wA?t=1768
Jun 03 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/20 4:02 PM, Walter Bright wrote:

 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:

 tbh I think this case looks worse than it is.


 
 Not if you look at the code generated by:
 
    import std.stdio;
 
    void test()
    {
      writeln("hello");
    }
 
 It's embarrassing.


Here it is: https://godbolt.org/z/LjUtc4

I was surprised.

It would be great if someone took a look at that.
Jun 05 2020
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:

 On 6/3/20 4:02 PM, Walter Bright wrote:

 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:

 tbh I think this case looks worse than it is.


 
 Not if you look at the code generated by:
 
    import std.stdio;
 
    void test()
    {
      writeln("hello");
    }
 
 It's embarrassing.


 Here it is: https://godbolt.org/z/LjUtc4

 I was surprised.

 It would be great if someone took a look at that.


What surprised you?
Jun 05 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/5/20 4:12 PM, Stefan Koch wrote:

 On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:

 On 6/3/20 4:02 PM, Walter Bright wrote:

 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:

 tbh I think this case looks worse than it is.


 Not if you look at the code generated by:

    import std.stdio;

    void test()
    {
      writeln("hello");
    }

 It's embarrassing.


 Here it is: https://godbolt.org/z/LjUtc4

 I was surprised.

 It would be great if someone took a look at that.


 
 What surprised you?


Code, code everywhere...
Jun 05 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 5 June 2020 at 20:18:01 UTC, Andrei Alexandrescu wrote:

 On 6/5/20 4:12 PM, Stefan Koch wrote:

 On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu 
 wrote:

 On 6/3/20 4:02 PM, Walter Bright wrote:

 On 6/3/2020 7:04 AM, Adam D. Ruppe wrote:

 tbh I think this case looks worse than it is.


 Not if you look at the code generated by:

    import std.stdio;

    void test()
    {
      writeln("hello");
    }

 It's embarrassing.


 Here it is: https://godbolt.org/z/LjUtc4

 I was surprised.

 It would be great if someone took a look at that.


 
 What surprised you?


 Code, code everywhere...


To me it looks like an accurate translation of the source code.
Jun 05 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:

 It would be great if someone took a look at that.


Yeah, this is why when reducing bugs I often prefer to use 
`printf` or even `assert(0)` or `asm { int 3; }` since it does 
result in a lot less code generated.

BUT, in the context of a regular program, I caution against 
making too big a deal of this. Yes, there's more code than there 
needs to be *for this simple case*, but much of it exists to 
support real world code that is more complicated out of necessity.

People don't actually write "hello world" programs. It'd be 
amusing to have the compiler recognize the input pattern and 
output some hand-tuned perfectly assembly for it.... but it'd 
also be useless. Once you start writing slices instead of 
literals, or enums and structs instead of basic types, much of 
that code is useful anyway.

Not all of it, again, I know some of writeln's branches are 
horrible, and I personally think the wrapping through C's lib 
with additional thread safety on top of it is silly. But those 
decisions are all made for debatably good reasons and at the end 
of the day, is it worth worrying about 0.04s and 200 bytes on a 
hello world build?
Jun 05 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 05, 2020 at 08:41:53PM +0000, Adam D. Ruppe via Digitalmars-d wrote:

 On Friday, 5 June 2020 at 20:10:30 UTC, Andrei Alexandrescu wrote:

 It would be great if someone took a look at that.


 
 Yeah, this is why when reducing bugs I often prefer to use `printf` or
 even `assert(0)` or `asm { int 3; }` since it does result in a lot
 less code generated.


I guess YMMV and all that, but I find all the "unnecessary
complications" of writefln eminently useful when debugging, esp. with
the %(...%) format specs that can print ranges and can be nested. Very
useful for one-line, throwaway debug dumps of range contents that
doesn't require writing an entire function and range-interpretation
framework just to get some miserable debug output, like you need to in
C. With write(f)ln's "complications", you can just write a 1-line debug
dump that you throw away after the next code-compile-debug cycle without
thinking twice.



 BUT, in the context of a regular program, I caution against making too
 big a deal of this. Yes, there's more code than there needs to be *for
 this simple case*, but much of it exists to support real world code
 that is more complicated out of necessity.


Yes, we're blowing so much steam over a measly hello world program when
larger, much more important issues lie unsolved.  Typical bikeshed vs.
nuclear reactor situation. :-D



 People don't actually write "hello world" programs. It'd be amusing to
 have the compiler recognize the input pattern and output some
 hand-tuned perfectly assembly for it.... but it'd also be useless.


[...]

Even more amusing would be for the compiler to recognize the pattern,
and launch a browser to some tutorial page on dlang.org explaining next
steps to go after your hello world program runs. :-P

But yeah, also equally useless.


T

-- 
The peace of mind---from knowing that viruses which exploit Microsoft system
vulnerabilities cannot touch Linux---is priceless. -- Frustrated system
administrator.
Jun 05 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 5 June 2020 at 21:07:26 UTC, H. S. Teoh wrote:

 I guess YMMV and all that, but I find all the "unnecessary 
 complications" of writefln eminently useful when debugging,


oh i agree, I like it when going through that process. writefln 
is p nice.

just at the *end* of the process, when it comes down to reducing 
the bug to the smallest thing so we can inspect the disassembly 
and report it upstream, then I avoid it.
Jun 05 2020
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 05, 2020 at 09:18:20PM +0000, Adam D. Ruppe via Digitalmars-d wrote:

 On Friday, 5 June 2020 at 21:07:26 UTC, H. S. Teoh wrote:

 I guess YMMV and all that, but I find all the "unnecessary
 complications" of writefln eminently useful when debugging,


 
 oh i agree, I like it when going through that process. writefln is p
 nice.
 
 just at the *end* of the process, when it comes down to reducing the
 bug to the smallest thing so we can inspect the disassembly and report
 it upstream, then I avoid it.


Well, if you're after the disassembly, assert(0) seems to be the ticket.
:-)


T

-- 
One reason that few people are aware there are programs running the internet is
that they never crash in any significant way: the free software underlying the
internet is reliable to the point of invisibility. -- Glyn Moody, from the
article "Giving it all away"
Jun 05 2020
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/5/2020 1:41 PM, Adam D. Ruppe wrote:

 But those decisions are all made for debatably good reasons 
 and at the end of the day, is it worth worrying about 0.04s and 200 bytes on a 
 hello world build?


writeln() is a fundamental piece of Phobos, and should be done right.

Back in the olden days, Borland's C compiler didn't produce very good code. But 
Borland managed to still get good results by doing the basics right - they went 
all out on hand-optimizing the C standard library.

In particular, they implemented printf entirely in hand-optimized assembler. 
Since printf is used everywhere in C code, this paid of handsomely for Borland. 
AFAIK, they're the only C vendor that did this.
Jun 05 2020
prev sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:


 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.

 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.


A good `writeln` would not call into C  at all. Nor would it copy 
its arguments like four times (therefore potentially not printing 
what it was given). But Phobos' `writeln` is actually a 
`encumber_you_with_dependencies_and_thread_safety_because_it_must_be_good_for_you_
oundabout_writeln`. So it's OK, we're only missing a plain `writeln`.
Jun 03 2020
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 3 June 2020 at 14:58:59 UTC, Stanislav Blinov wrote:

 On Wednesday, 3 June 2020 at 03:19:37 UTC, Walter Bright wrote:


 The thing is, `writeln("hello")` can be expanded to 
 `core.stdc.stdio.puts("hello")`. Done.

 Phobos seems to do a lot of "going around the Horn" instead of 
 taking the canal.


 A good `writeln` would not call into C  at all. Nor would it 
 copy its arguments like four times (therefore potentially not 
 printing what it was given). But Phobos' `writeln` is actually 
 a 
 `encumber_you_with_dependencies_and_thread_safety_because_it_must_be_good_for_you_
oundabout_writeln`. So it's OK, we're only missing a plain `writeln`.


To be fair, C's `puts` also includes thread-safety. You need to 
use non-standard functions like `fputs_unlocked` if you want to 
avoid that overhead.
Jun 03 2020
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 7:58 AM, Stanislav Blinov wrote:

 A good `writeln` would not call into C  at all.


C's puts() is thread safe. No reason to not use it.



 So it's OK, we're only missing a plain `writeln`.


Doing the basic, common cases by going around the horn is not good engineering.
Jun 03 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 20:05:39 UTC, Walter Bright wrote:

 On 6/3/2020 7:58 AM, Stanislav Blinov wrote:

 A good `writeln` would not call into C  at all.


 C's puts() is thread safe. No reason to not use it.


A thread-safe writeln would be threadsafe_writeln. Or, 
alternatively there would be a writeln and a 
not_threadsafe_writeln. No reason to pay for something you're not 
using. And still neither implementation would call into C, the OS 
is right there.
I guess this is where, in Phobos, one could use the output range 
format API with a custom sink. But that wouldn't help against all 
the unnecessary copying.


 So it's OK, we're only missing a plain `writeln`.


 Doing the basic, common cases by going around the horn is not 
 good engineering.


On that, we're in agreement.
Jun 03 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 4:29 PM, Stanislav Blinov wrote:

 On Wednesday, 3 June 2020 at 20:05:39 UTC, Walter Bright wrote:

 On 6/3/2020 7:58 AM, Stanislav Blinov wrote:

 A good `writeln` would not call into C  at all.


 C's puts() is thread safe. No reason to not use it.


 
 A thread-safe writeln would be threadsafe_writeln. Or, alternatively 
 there would be a writeln and a not_threadsafe_writeln. No reason to pay 
 for something you're not using. And still neither implementation would 
 call into C, the OS is right there.


writeln writes to a FILE *, which is C. There's no way around it. FILE * 
is buffered i/o, so going around it would mean odd out-of-order output.

-Steve
Jun 03 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 21:04:23 UTC, Steven Schveighoffer 
wrote:


 A thread-safe writeln would be threadsafe_writeln. Or, 
 alternatively there would be a writeln and a 
 not_threadsafe_writeln. No reason to pay for something you're 
 not using. And still neither implementation would call into C, 
 the OS is right there.


 writeln writes to a FILE *, which is C. There's no way around 
 it. FILE * is buffered i/o, so going around it would mean odd 
 out-of-order output.


There's no way around writing to a FILE*? I... hope you didn't 
mean what you wrote, all things considered ;)

I know what Phobos' writeln does. But whatever, this is not the 
topic. Maybe in another thread at another time...
Jun 03 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 5:29 PM, Stanislav Blinov wrote:

 On Wednesday, 3 June 2020 at 21:04:23 UTC, Steven Schveighoffer wrote:
 

 A thread-safe writeln would be threadsafe_writeln. Or, alternatively 
 there would be a writeln and a not_threadsafe_writeln. No reason to 
 pay for something you're not using. And still neither implementation 
 would call into C, the OS is right there.


 writeln writes to a FILE *, which is C. There's no way around it. FILE 
 * is buffered i/o, so going around it would mean odd out-of-order output.


 
 There's no way around writing to a FILE*? I... hope you didn't mean what 
 you wrote, all things considered ;)


Well, no way around it if you want writeln to do expected things ;)

Unless you are suggesting that Phobos completely drop support for 
interleaving printf and writeln calls? If you are, I assure you Walter 
will not accept that.


 I know what Phobos' writeln does. But whatever, this is not the topic. 
 Maybe in another thread at another time...


Not sure why you think the implementation of Phobos using C for I/O is 
not on-topic. It was you who suggested we could avoid it for certain 
calls to writeln.

-Steve
Jun 03 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 3 June 2020 at 22:21:41 UTC, Steven Schveighoffer 
wrote:

 On 6/3/20 5:29 PM, Stanislav Blinov wrote:



 There's no way around writing to a FILE*? I... hope you didn't 
 mean what you wrote, all things considered ;)


 Well, no way around it if you want writeln to do expected 
 things ;)

 Unless you are suggesting that Phobos completely drop support 
 for interleaving printf and writeln calls? If you are, I assure 
 you Walter will not accept that.


Phobos can have whatever it has. That wasn't my point.


 Not sure why you think the implementation of Phobos using C for 
 I/O is not on-topic. It was you who suggested we could avoid it 
 for certain calls to writeln.


Because topic is template bloat, err... statistics :) And I'm 
well at fault for straying.
I was speaking figuratively. A *good* writeln need not go to C - 
that was my retort to bringing up puts in 2020. Not *the* 
writeln, *a* writeln. At the very least it shouldn't be the 
default. My point is we should be striving for better, not that 
there's anything wrong with puts or C.
Jun 03 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 9:27 PM, Stanislav Blinov wrote:

 On Wednesday, 3 June 2020 at 22:21:41 UTC, Steven Schveighoffer wrote:

 On 6/3/20 5:29 PM, Stanislav Blinov wrote:


 

 There's no way around writing to a FILE*? I... hope you didn't mean 
 what you wrote, all things considered ;)


 Well, no way around it if you want writeln to do expected things ;)

 Unless you are suggesting that Phobos completely drop support for 
 interleaving printf and writeln calls? If you are, I assure you Walter 
 will not accept that.


 
 Phobos can have whatever it has. That wasn't my point.


Ah, I get it, sorry for misunderstanding of your point. Yes, I'm in the 
same boat. I've hated how Phobos uses FILE * for its I/O.


 

 Not sure why you think the implementation of Phobos using C for I/O is 
 not on-topic. It was you who suggested we could avoid it for certain 
 calls to writeln.


 
 Because topic is template bloat, err... statistics :) And I'm well at 
 fault for straying.
 I was speaking figuratively. A *good* writeln need not go to C - that 
 was my retort to bringing up puts in 2020. Not *the* writeln, *a* 
 writeln. At the very least it shouldn't be the default. My point is we 
 should be striving for better, not that there's anything wrong with puts 
 or C.


Agree on all that, sorry for the confusion.

-Steve
Jun 03 2020
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 3:21 PM, Steven Schveighoffer wrote:

 Unless you are suggesting that Phobos completely drop support for interleaving 
 printf and writeln calls? If you are, I assure you Walter will not accept that.


I just checked with him, and he assured me he won't.

Also, a plain writeln("hello") must be thread safe just like puts("hello"). The 
reason is simply customer demand. I recall the bad old days when multiple 
threads would write to stdout, and you'd get:

   heworllldo

and everyone would complain. People accept that printfs can be interleaved on 
the function call level, but not inside the call. Furthermore, stdout has to 
acquire a mutex before writing to it:

     acquire stdout mutex
     print 'h'
     release stdout mutex
     acquire stdout mutex
     print 'e'
     release stdout mutex
     acquire stdout mutex
     print 'l'
     release stdout mutex

turns out to be disastrously slower than:

     acquire stdout mutex
     print 'hello'
     release stdout mutex
Jun 03 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 4 June 2020 at 01:38:50 UTC, Walter Bright wrote:

 Furthermore, stdout has to acquire a mutex before writing to it:


No need for any of this on the OS level, a write up to like 4k is 
guaranteed to be atomic.

It is the C buffer that isn't :(

And the C buffer is a consistent source of pain for 
user-interactive i/o. Buffering makes a big performance 
difference when you are actually outputting a block... but with 
user-interactive you rarely actually *want* buffering since users 
want to see their results ASAP! And remember, pipes are 
frequently user-interactive too and them not being flushed 
frequently is a FAQ among new users, especially IDE users.

I kinda feel buffering should not be done across function 
boundaries for stdout and stderr. Other files maybe.
Jun 03 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 10:28 PM, Adam D. Ruppe wrote:

 On Thursday, 4 June 2020 at 01:38:50 UTC, Walter Bright wrote:

 Furthermore, stdout has to acquire a mutex before writing to it:


 
 No need for any of this on the OS level, a write up to like 4k is 
 guaranteed to be atomic.
 
 It is the C buffer that isn't :(
 
 And the C buffer is a consistent source of pain for user-interactive 
 i/o. Buffering makes a big performance difference when you are actually 
 outputting a block... but with user-interactive you rarely actually 
 *want* buffering since users want to see their results ASAP! And 
 remember, pipes are frequently user-interactive too and them not being 
 flushed frequently is a FAQ among new users, especially IDE users.
 
 I kinda feel buffering should not be done across function boundaries for 
 stdout and stderr. Other files maybe.


I don't think you are consider the ramifications. You NEED a buffer, 
otherwise, e.g. you will be issuing system calls for writing 5 bytes, 
followed by an immediate system call for 5 more. And that isn't cheap. 
Locking a mutex would be dwarfed by that. Flushing on newlines is a 
reasonable compromise.

But it doesn't have to be a C buffer (unless you want to interact with 
printf).

There are good reasons to use C buffering for the standard handles. But 
it sucks that ALL Phobos I/O has to be based on it (including opening a 
file that you never use in a C call).

-Steve
Jun 03 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 4 June 2020 at 03:00:26 UTC, Steven Schveighoffer 
wrote:

 buffer, otherwise, e.g. you will be issuing system calls for 
 writing 5 bytes, followed by an immediate system call for 5 
 more.


Yeah, I know. You and I have both written low-level I/O code, we 
both know how it works and the huge difference proper buffering 
can make.

But the question is where do you have implicit flushes? I'm 
arguing for at the end of the function.

The call to writefln or whatever would do an intermediate buffer, 
but when it returns, it flushes it all at once. You don't flush 
intermediates if you can get away with it, but you treat each 
function as a direct write.

If a user actually writes out `write("h"); write("e"); 
write("l");`, well, that's on them, maybe they wanted to see it 
spell out!

But if they write `writefln("hello, %s, you have %d points!", 
name, score);`, the implementation should NOT do

os.write("hello, ");
os.write(name);
os.write(", you have ");

no, it should go ahead and buffer those pieces, then at the very 
end write it all.

Just again, if the USER writes it out in separate calls... well, 
that's on them.


This will match what happens in most cases anyway since the C 
buffer DOES flush on \n most the time (and when it doesn't, users 
complain and I have to explain it to them AGAIN)


getting a lil off topic for templates btw.......
Jun 03 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/3/20 11:14 PM, Adam D. Ruppe wrote:


 getting a lil off topic for templates btw.......


Yeah, for sure.

My one final point would be that sometimes writing things out piecemeal 
based on logic is natural. I don't think the user should have to do 
acrobatics to fit it all in one call.

BTW, I misunderstood your original point, and thought you meant stdout 
and stderr should not be buffered at all.

-Steve
Jun 03 2020
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/3/2020 7:28 PM, Adam D. Ruppe wrote:

 I kinda feel buffering should not be done across function boundaries for
stdout 
 and stderr. Other files maybe.


That's why there's "isatty()".
Jun 03 2020
prev sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Thursday, 4 June 2020 at 01:38:50 UTC, Walter Bright wrote:

 On 6/3/2020 3:21 PM, Steven Schveighoffer wrote:

 Unless you are suggesting that Phobos completely drop support 
 for interleaving printf and writeln calls? If you are, I 
 assure you Walter will not accept that.


 I just checked with him, and he assured me he won't.


:)


 Also, a plain writeln("hello") must be thread safe just like 
 puts("hello"). The reason is simply customer demand. I recall 
 the bad old days when multiple threads would write to stdout, 
 and you'd get:

   heworllldo


That's simple, multiple threads shouldn't be writing to stdout ;) 
And that's only a half-joke. There of course must be a 
thread-safe writeln, just as there must be a non-thread-safe 
writeln (with different names, obviously). Which is the default 
should depend on the program. If you're only running one thread, 
or more to the point, only doing stdout from one thread [at a 
time], you have no use for those kinds of locks. It's just waste 
of cache. If you're not using any C libraries, or ones you are 
using aren't doing stdout, you have no use for FILE*. The OS is 
right there, a syscall away.

Phobos does make provisions for rolling your own sink 
(formattedWrite), which is great (more templates though :P). But, 
when you do sit down to write your own, core.sys is startlingly 
deprived of sys and is full of C. And now it becomes a roll your 
own syscalls... And then you find core.osthread full of pthread 
and not osthread.
Jun 03 2020
prev sibling parent Max Samukha <maxsamukha gmail.com> writes:
On Wednesday, 3 June 2020 at 20:05:39 UTC, Walter Bright wrote:

 On 6/3/2020 7:58 AM, Stanislav Blinov wrote:

 A good `writeln` would not call into C  at all.


 C's puts() is thread safe. No reason to not use it.



 So it's OK, we're only missing a plain `writeln`.


 Doing the basic, common cases by going around the horn is not 
 good engineering.


Superfluous special cases and premature optimization is not good 
software engineering either.
Jun 03 2020
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 3 June 2020 at 14:58:59 UTC, Stanislav Blinov wrote:


 A good `writeln` would not call into C  at all.


What would it do then? Call syscalls in the kernel directly? 
That's more or less only supported on Linux. It's definitely not 
supported on macOS. You can do it, but it's not supported and 
make break at any time. Go used to do that, something broke for 
them and now they're going through the C wrappers.

--
/Jacob Carlborg
Jun 04 2020
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Thursday, 4 June 2020 at 09:10:06 UTC, Jacob Carlborg wrote:

 On Wednesday, 3 June 2020 at 14:58:59 UTC, Stanislav Blinov 
 wrote:


 A good `writeln` would not call into C  at all.


 What would it do then? Call syscalls in the kernel directly? 
 That's more or less only supported on Linux. It's definitely 
 not supported on macOS. You can do it, but it's not supported 
 and make break at any time. Go used to do that, something broke 
 for them and now they're going through the C wrappers.


Going through the system's library to *make syscalls*, yes. 
Obviously I'm presenting an idealized case. If you can't sustain 
direct interop with the OS to maintain compatibility, then yes, 
while unfortunate, you'd use what you have to. But that is a far 
cry from unconditionally routing your stdout through FILE*.
The situation with Mac only highlights how infectuous C actually 
is. And Linux is only slightly better, to be honest.
Jun 05 2020
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 3 June 2020 at 02:32:54 UTC, Walter Bright wrote:

 On 6/2/2020 5:53 AM, Ethan wrote:

 Oh I have *got* to try this out on my code.


 I tried it on `writeln("hello");` and the number of templates 
 instantiated seems excessive. It should be investigated.

   Number   Unique   Name
       46       38   MAKEINTRESOURCE_T(ushort i)
       38        0   OldAlias(T) if (!isAggregateType!T || 
 is(Unqual!T == T))
        7        6   staticIndexOf(T, TList...)
        1        1   Impl(T)
        7        5   isSomeChar(T)
        2        1   StringTypeOf(T)
        1        1   Impl(T)
        8        2   mostNegative(T) if (isNumeric!T || 
 isSomeChar!T || isBoolean!T)
        1        1   encode(Flag useReplacementDchar = 
 No.useReplacementDchar)(out char[4] buf, dchar c)
        7        7   ModifyTypePreservingTQ(alias Modifier, T)
        2        1   Unqual(T)
        1        1   put(A)(A writeme) if 
 ((isSomeChar!(Unqual!(ElementType!A)) || is(ElementType!A : 
 const(ubyte))) && i
 sInputRange!A && !isInfinite!A)
        1        1   decodeImpl(bool canIndex, Flag 
 useReplacementDchar = No.useReplacementDchar, S)(auto ref S 
 str, ref
 size_t index) if (is(S : const(char[])) || isInputRange!S && 
 is(Unqual!(ElementEncodingType!S) == char))
        3        1   isCallable(T...) if (T.length == 1)
        1        1   trustedFwrite(T)(FILE* f, const T[] obj)
        2        1   _utfException(Flag 
 useReplacementDchar)(string msg, dchar c)
        1        1   popFront(C)(ref C[] str) if (Autodecoding 
 && isNarrowString!(C[]))
        2        2   IntegralTypeOf(T)
        1        1   Impl(T)
        1        1   Impl(T)
       20       20   genericIndexOf(args...) if (args.length >= 
 1)
        1        1   safeOp(T0, T1)(auto ref T0 a, auto ref T1 b)
        5        1   ElementEncodingType(R)
        1        1   errnoEnforce(T, string file = __FILE__, 
 uint line = __LINE__)(T value, lazy string msg = null)
        1        1   Impl(T)
        2        2   put(C)(C c) if (isSomeChar!C || is(C : 
 const(ubyte)))
        2        1   exception(S)(S str, string msg)
        6        1   isNarrowString(T)
        2        1   at(R)(R[] r, size_t i)
        1        1   FunctionTypeOf(func...) if (func.length == 
 1 && isCallable!func)
        1        1   MinType(T...) if (T.length >= 1)
        5        0   AliasSeq(TList...)
        4        1   opDispatch(string name)
        1        1   min(T...)(T args) if (T.length >= 2)
        8        8   sndAlias(char c0, char c1)
        1        1   __equals(T1, T2)(T1[] lhs, T2[] rhs)
       55       13   isAggregateType(T)
        8        2   isNumeric(T)
        7        7   OriginalType(T)
        4        1   isInputRange(R)
        1        1   empty(T)(in T[] a)
        6        3   ElementType(R)
       36       11   expectType(T)
       24        7   Unqual(T)
        1        1   MAKEINTATOM_T()(int i)
        1        1   writeln(T...)(T args)
        1        1   Impl(T)
        3        1   isSomeString(T)
        1        1   isSomeFunction(T...) if (T.length == 1)
       15        2   TypeDef(T)
        1        1   isInfinite(R)
        3        1   isStaticArray(T)
       18       18   isSame(ab...) if (ab.length == 2)
        2        1   front(T)(T[] a) if (Autodecoding && 
 isNarrowString!(T[]))
        1        1   decode(Flag useReplacementDchar = 
 No.useReplacementDchar, S)(auto ref S str, ref size_t index) if 
 (i
 sSomeString!S)
        1        1   codeUnitLimit(S) if 
 (isSomeChar!(ElementEncodingType!S))
        2        1   Parameters(func...) if (func.length == 1 && 
 isCallable!func)
        1        1   fd_set_custom(uint SETSIZE)
        4        4   mmioFOURCC(char c0, char c1, char c2, char 
 c3)
        2        1   Demangle(T)
        4        3   Flag(string name)
        1        1   safeOp(string S) if (S == "<" || S == ">" 
 || S == "<=" || S == ">=" || S == "==" || S == "!=")
        2        2   isIntegral(T)
        1        1   Impl(T)
        5        5   CharTypeOf(T)


looks like abstractions are not zero cost in fine
Jun 02 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 03:06:37 UTC, Basile B. wrote:

 On Wednesday, 3 June 2020 at 02:32:54 UTC, Walter Bright wrote:

        [...]


 looks like abstractions are not zero cost in fine


Looks like you haven't used `-vcg-ast` either.
If you do you'll see what's going on.

Some of these can be replaced with CTFE.
Generally only things that need to work on template parameter 
lists are forced to be templates.

Well and legitimate type parameterization of course.
But those are the minority.

If the template sounds like a question about a type,
Like (CharTypeof or isIntegral) it probably should not be a 
template.
Jun 02 2020
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 3 June 2020 at 04:52:37 UTC, Stefan Koch wrote:

 [snip]

 If the template sounds like a question about a type,
 Like (CharTypeof or isIntegral) it probably should not be a 
 template.



I was a little confused by this.

Given your recent work on type functions, do you mean that type 
functions should replace templates in situations like that? That 
would imply that things like `isIntegral` has to be a template 
now, but could be changed in the future, correct? Do you think 
something like mir's `sumType` [1] should not be a template?


[1] 
https://github.com/libmir/mir-algorithm/blob/f111c311eadb0057f5a233c156b772a010ceab71/source/mir/math/sum.d#L1986
Jun 03 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 11:09:56 UTC, jmh530 wrote:

 On Wednesday, 3 June 2020 at 04:52:37 UTC, Stefan Koch wrote:

 [snip]

 If the template sounds like a question about a type,
 Like (CharTypeof or isIntegral) it probably should not be a 
 template.



 I was a little confused by this.

 Given your recent work on type functions, do you mean that type 
 functions should replace templates in situations like that? 
 That would imply that things like `isIntegral` has to be a 
 template now, but could be changed in the future, correct? Do 
 you think something like mir's `sumType` [1] should not be a 
 template?


 [1] 
 https://github.com/libmir/mir-algorithm/blob/f111c311eadb0057f5a233c156b772a010ceab71/source/mir/math/sum.d#L1986


Oh yes.
sumType does not need to introduce new symbols.
Therefore it should be handled in a way which does not introduce 
new symbols.

type functions are one way to make this happen.
However changes to the language are hard justify therefore I am 
currently revisiting,
approaches to infer type-function-like behavior and transform 
them behind the scenes.
(Which is a HUGE pain, because you have to transform recursion 
into loops, which is a hard problem in itself, AND the user 
probably would rather have written a loop, but was forced by the 
language to use recursion. )

alas having my work being useful in the language is more 
important than to have a better language.

I will continue on type functions; if the conservative approaches 
don't work out.
Jun 03 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:

 Oh yes.
 sumType does not need to introduce new symbols.
 Therefore it should be handled in a way which does not 
 introduce new symbols.


In other words, we want a way to give our templates 
AST-macro-like evaluation semantics. It's the same thing with 
Manu's `...` and staticMap. :)

I wonder if a DIP to add something like "Macros: the Good Parts" 
to D would be accepted?
Jun 03 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 3 June 2020 at 14:13:47 UTC, Paul Backus wrote:

 On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:

 Oh yes.
 sumType does not need to introduce new symbols.
 Therefore it should be handled in a way which does not 
 introduce new symbols.


 In other words, we want a way to give our templates 
 AST-macro-like evaluation semantics. It's the same thing with 
 Manu's `...` and staticMap. :)

 I wonder if a DIP to add something like "Macros: the Good 
 Parts" to D would be accepted?


I doubt that it would have any more success than type functions.
Jun 03 2020
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:

[snip]
 Oh yes.
 sumType does not need to introduce new symbols.
 Therefore it should be handled in a way which does not 
 introduce new symbols.

 type functions are one way to make this happen.
 However changes to the language are hard justify therefore I am 
 currently revisiting,
 approaches to infer type-function-like behavior and transform 
 them behind the scenes.
 (Which is a HUGE pain, because you have to transform recursion 
 into loops, which is a hard problem in itself, AND the user 
 probably would rather have written a loop, but was forced by 
 the language to use recursion. )

 alas having my work being useful in the language is more 
 important than to have a better language.

 I will continue on type functions; if the conservative 
 approaches don't work out.


Thanks.

The problem almost sounds like you need an inliner for types in 
the compiler.
Jun 03 2020
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:

 AND the user probably would rather have written a loop, but was 
 forced by the language to use recursion.


That's definitely true.


 I will continue on type functions; if the conservative 
 approaches don't work out.


I hope we can get type functions.

--
/Jacob Carlborg
Jun 04 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 4 June 2020 at 08:38:59 UTC, Jacob Carlborg wrote:

 On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:

 AND the user probably would rather have written a loop, but 
 was forced by the language to use recursion.


 That's definitely true.


 I will continue on type functions; if the conservative 
 approaches don't work out.


 I hope we can get type functions.

 --
 /Jacob Carlborg


Well to be completely honest type functions do look cool.
But they to have a their own set of problems.
I designed them so it would be impossible to "abuse" them.
I did that by putting a restriction in that would disallow them 
from generating symbols.
And that does work and greatly simplifies the implementation.
However, it also so means that they can't be used in many places 
that you would like to use them.
For example creating a std.typecons.Tuple is impossible with type 
functions alone.
For such a task they can only be used to reduce the intermediate 
symbols within a template.
Which I fear will lead to some frustration.
Also you would need to rewrite your entire reflection framework 
to use typefunctions.
For that to be feasible new __traits have to be introduced.
.....

All in all even though the feature alone is not that heavy, it 
will need major additions to be useful.
I cannot do this alone.
The way I see it; If Walter is not in the boat the boat will not 
get anywhere.
It's to heavy for me to move by myself.
Jun 04 2020
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 3 June 2020 at 11:22:23 UTC, Stefan Koch wrote:

 [snip]
 type functions are one way to make this happen.
 However changes to the language are hard justify therefore I am 
 currently revisiting,
 approaches to infer type-function-like behavior and transform 
 them behind the scenes.
 (Which is a HUGE pain, because you have to transform recursion 
 into loops, which is a hard problem in itself, AND the user 
 probably would rather have written a loop, but was forced by 
 the language to use recursion. )
 [snip]


I was just thinking about this again in light of Manu's recent 
post [1] on the bonus thread. Manu specifically mentions types 
not working well in static contexts like types and aliases.

It occurs to me that there may be a connection between what Manu 
said and the issue that DIP 1023 [2] attempted to resolve. For 
instance, consider the DIP example
alias PackedUpperTriangularMatrix(T) = Slice!(StairsIterator!(T*, 
"-"));
What if instead you have a function that takes a type and returns 
an alias to another type? Ignoring any issues with the syntax, it 
might look something like this
alias PackedUpperTriangularMatrix(T)() {
     return alias Slice!(StairsIterator!(T*, "-"));
}
and could be used like
auto foo(T)(PackedUpperTriangularMatrix!T m) { }
(which is the same as in the DIP).

At the end of the day, you would still need to ensure the 
compiler would actually resolve those types before the function 
is called. So I imagine that just adding the support to be able 
to do the first bit does not imply the second bit would work. 
However, the issue with the solution presented in DIP 1023 was 
that it was only designed to work with the short alias syntax and 
not the full template alias syntax, which may not have been 
possible and could have led to a special casing. This is a simple 
rule: resolve the type/alias functions first.

I know on other threads you have talked about how much work it 
would be to go further with type functions and you would need 
Walter's blessing and help. However, I think that if it means 
that we could get an alternative resolution to the issues brought 
up by DIP 1023, then I would be all on board (and willing to 
contribute to support the work).


[1] 
https://forum.dlang.org/post/mailman.3585.1591274099.31109.digitalmars-d puremagic.com
[2] 
https://github.com/dlang/DIPs/blob/master/DIPs/other/DIP1023.md
Jun 04 2020
prev sibling next sibling parent Atwork <fwafwa fwfafa.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


This makes me happy, thank you!
Jun 03 2020
prev sibling next sibling parent James Lu <jamtlu gmail.com> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


What if DMD had an opt-in option to gather and send detailed 
profiling data on compiles automatically? (If someone wanted to 
they could use Differential Privacy to randomize the numbers 
while preserving the overall informational content.)
Jun 05 2020
prev sibling parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are blowing 
 up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


Looks super useful.

I tried it on the test suite of openmethods and I see lots of 
duplicate entries, e.g.:

[...]
        2        1   setName(NewValue...)
        5        5   UnpackDeepImpl(U...)
      121       40   Parameters(func...) if (func.length == 1 && 
isCallable!func)
        8        8   UnpackDeepImpl(U...)
        2        2   argumentMixtureAt(int i)
        3        3   UnpackDeepImpl(U...)
        4        4   UnpackDeepImpl(U...)
        5        5   UnpackDeepImpl(U...)
[...]
Jun 06 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 6 June 2020 at 12:54:11 UTC, Jean-Louis Leroy wrote:

 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


 Looks super useful.

 I tried it on the test suite of openmethods and I see lots of 
 duplicate entries, e.g.:

 [...]
        2        1   setName(NewValue...)
        5        5   UnpackDeepImpl(U...)
      121       40   Parameters(func...) if (func.length == 1 && 
 isCallable!func)
        8        8   UnpackDeepImpl(U...)
        2        2   argumentMixtureAt(int i)
        3        3   UnpackDeepImpl(U...)
        4        4   UnpackDeepImpl(U...)
        5        5   UnpackDeepImpl(U...)
 [...]


Duplicate entries mean different Template Declarations with the 
same name.
Usually those happen when templates are nested.

Consider:
template X(uint I1)
{
     template Y()
     {
        alias Y = I1;
     }
     alias X = Y!();
}

In this case the inner declaration is not dependent on any 
parameter that was passed to it.
But it is dependent on the context in which it is instantiated.
Every Instance of X will create a new separate context, In which 
a unique declaration of Y is formed.
Jun 06 2020
parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Saturday, 6 June 2020 at 13:23:38 UTC, Stefan Koch wrote:

 On Saturday, 6 June 2020 at 12:54:11 UTC, Jean-Louis Leroy 
 wrote:

 On Tuesday, 2 June 2020 at 06:47:54 UTC, Walter Bright wrote:

 Many people are trying to figure out what templates are 
 blowing up their compile. This should help:

 https://github.com/dlang/dmd/pull/11208


 Looks super useful.

 I tried it on the test suite of openmethods and I see lots of 
 duplicate entries, e.g.:

 [...]
        2        1   setName(NewValue...)
        5        5   UnpackDeepImpl(U...)
      121       40   Parameters(func...) if (func.length == 1 
 && isCallable!func)
        8        8   UnpackDeepImpl(U...)
        2        2   argumentMixtureAt(int i)
        3        3   UnpackDeepImpl(U...)
        4        4   UnpackDeepImpl(U...)
        5        5   UnpackDeepImpl(U...)
 [...]


 Duplicate entries mean different Template Declarations with the 
 same name.
 Usually those happen when templates are nested.

 Consider:
 template X(uint I1)
 {
     template Y()
     {
        alias Y = I1;
     }
     alias X = Y!();
 }

 In this case the inner declaration is not dependent on any 
 parameter that was passed to it.
 But it is dependent on the context in which it is instantiated.
 Every Instance of X will create a new separate context, In 
 which a unique declaration of Y is formed.


I suspected something like this.

I am trying to aggregate the template stats across several 
translation units (in fact the test suite of openmethods), to 
measure whether an implementation is better or worse than another 
of the same thing. In your example, what happens if I make a copy 
of X, name it Z, but leave the body unchanged? I won't be able to 
distinguish the X.Ys from the Z.Ys. Even within the same 
template, imagine I put different definitions of Y inside a 
`static if else`. I could not tell how many times each was used. 
So maybe template stats should display the fully qualified name 
(including the module) along with the line and column numbers.

Maybe I'm missing something, I will try to find time to 
experiment more during the week-end.
Jun 06 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 6 June 2020 at 13:59:38 UTC, Jean-Louis Leroy wrote:

 On Saturday, 6 June 2020 at 13:23:38 UTC, Stefan Koch wrote:

 [...]


 I suspected something like this.

 I am trying to aggregate the template stats across several 
 translation units (in fact the test suite of openmethods), to 
 measure whether an implementation is better or worse than 
 another of the same thing. In your example, what happens if I 
 make a copy of X, name it Z, but leave the body unchanged? I 
 won't be able to distinguish the X.Ys from the Z.Ys. Even 
 within the same template, imagine I put different definitions 
 of Y inside a `static if else`. I could not tell how many times 
 each was used. So maybe template stats should display the fully 
 qualified name (including the module) along with the line and 
 column numbers.

 Maybe I'm missing something, I will try to find time to 
 experiment more during the week-end.


Adding the location is a good idea.
I am going to open a PR.
Fully qualified names however can absolutely gigantic for 
pathological cases, and I'd like to avoid that.
Jun 06 2020