www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - druntime, templates and traits

reply "monarch_dodra" <monarchdodra gmail.com> writes:
One of the "issues" of druntime is that is that almost everything 
in it is resolved at runtime. This is good because it avoids 
bloat. However, it also means we run into 2 regular issues:

1. Performance (small issue). A lot of code that should be 
otherwise fast is slower than it could be, since druntime will 
check, at runtime, if postblit is needed, if destruction should 
be done. For example, if you "dup" a string by hand, you can do 
it 50% faster than what druntime does.

2. Inference. This is an odd one. Currently, the function that 
operate on arrays (reserve, dup, length) etc... may or may not be 
 safe or pure. Or nothrow. The problem is that we really can't do 
anything about it. duping a string is obviously nothrow, but if 
you dup something with a postblit, then it might not be (nor 
safe).

These issues kind of keep popping up in phobos, and we will have 
to address them sooner or later. The current state of affairs is 
a kind of status quo of "its wrong, inconsistent, but maybe OK 
most of the time".

For example, "dup" is not nothrow, yet it is safe and pure. 
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may 
actually call postblit, but not assumeSafeAppend.

========

The solution (I think) would be to partially templatize 
d-runtime. We can keep most of what is in rt/lifetime.d, but 
statically split it into smaller chunks, which could be piloted 
with more accuracy via a templated "manager".

For now, if we could make a change as simple as "no postblit => 
safe pure nothrow, and fast guaranteed"/"postblit => not safe, 
not pure, may throw", then it would already be a big win (IMO).

========

I had started toying around with this, and the main and immediate 
challenge I ran into is the integration of templates. The 2 
issues are:
1. All templates must be in object.d, or the compiler won't see 
it.
2. A simple (and needed) trait like "hasElaborateCopyConstructor" 
requires a good third of both "typetuple.d" and "traits.d". 
"functionAttributes" is pretty simple though (I think).

So the two challenges are:
1. If we integrate a large amount of template code, where would 
it go?
2. How will we deal with not having massive code duplication?

========

I'm mostly looking to start a discussion on this, and for 
guidance, ideas, on where to start.
Sep 13 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
monarch_dodra:

 So the two challenges are:
 1. If we integrate a large amount of template code, where would 
 it go?

How much large it is?
 2. How will we deal with not having massive code duplication?

Go developers accept a little amount of duplication. But in your case a solution could be just to deprecate and later remove the Phobos code that you put in object.d. Bye, bearophile
Sep 13 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-09-13 12:26, monarch_dodra wrote:

 I had started toying around with this, and the main and immediate
 challenge I ran into is the integration of templates. The 2 issues are:
 1. All templates must be in object.d, or the compiler won't see it.

Do you know why that is? Can they be put in a separate module but imported by object.d?
 2. A simple (and needed) trait like "hasElaborateCopyConstructor"
 requires a good third of both "typetuple.d" and "traits.d".
 "functionAttributes" is pretty simple though (I think).

Can't we move some of the traits to druntime? std.traits would publicly import the ones which were moved.
 So the two challenges are:
 1. If we integrate a large amount of template code, where would it go?

It depends on where it needs to be used. Some suggestions: * core.traits|typetuple * rt.util.traits|typetuple
 2. How will we deal with not having massive code duplication?

Move to druntime? Phobos already depends on it. -- /Jacob Carlborg
Sep 13 2013
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Friday, 13 September 2013 at 10:26:35 UTC, monarch_dodra wrote:
 1. Performance (small issue). A lot of code that should be 
 otherwise fast is slower than it could be, since druntime will 
 check, at runtime, if postblit is needed, if destruction should 
 be done. For example, if you "dup" a string by hand, you can do 
 it 50% faster than what druntime does.

Yes, this is pitfall of D design.
 2. Inference. This is an odd one. Currently, the function that 
 operate on arrays (reserve, dup, length) etc... may or may not 
 be  safe or pure. Or nothrow. The problem is that we really 
 can't do anything about it. duping a string is obviously 
 nothrow, but if you dup something with a postblit, then it 
 might not be (nor safe).

Yes, this is pitfall of D design.
 These issues kind of keep popping up in phobos, and we will 
 have to address them sooner or later. The current state of 
 affairs is a kind of status quo of "its wrong, inconsistent, 
 but maybe OK most of the time".

They are systematically popping up because of systematical language features (D is statically typed language with little runtime supported).
 For example, "dup" is not nothrow, yet it is safe and pure. 
 Reserve is nothrow, yet assumeSafeAppend is not. Reserve may 
 actually call postblit, but not assumeSafeAppend.

 ========

 The solution (I think) would be to partially templatize 
 d-runtime. We can keep most of what is in rt/lifetime.d, but 
 statically split it into smaller chunks, which could be piloted 
 with more accuracy via a templated "manager".

It depends on how do you want to templatize it. Due to separate compilation model it is impossible to know beforehand (when you provide phobos library together with druntime) which types will be used.
 For now, if we could make a change as simple as "no postblit => 
 safe pure nothrow, and fast guaranteed"/"postblit => not safe, 
 not pure, may throw", then it would already be a big win (IMO).

 ========

 I had started toying around with this, and the main and 
 immediate challenge I ran into is the integration of templates. 
 The 2 issues are:
 1. All templates must be in object.d, or the compiler won't see 
 it.
 2. A simple (and needed) trait like 
 "hasElaborateCopyConstructor" requires a good third of both 
 "typetuple.d" and "traits.d". "functionAttributes" is pretty 
 simple though (I think).

Yes, this is pitfall of D design.
 I'm mostly looking to start a discussion on this, and for 
 guidance, ideas, on where to start.

I doubt that the problems you raised are solvable in general case with current runtime capabilities. While it still may be possible to tackle some of them, D design (separate compilation model, type system, templates, how pure, safe, nothrow are defined) would prevent you from fixing these problems.
Sep 13 2013
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Sep 13, 2013, at 5:35 AM, Jacob Carlborg <doob me.com> wrote:

 On 2013-09-13 12:26, monarch_dodra wrote:
=20
 I had started toying around with this, and the main and immediate
 challenge I ran into is the integration of templates. The 2 issues =


 1. All templates must be in object.d, or the compiler won't see it.

Do you know why that is? Can they be put in a separate module but =

Yes.
Sep 13 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Sep 13, 2013 at 03:20:10PM +0200, Maxim Fomin wrote:
 On Friday, 13 September 2013 at 10:26:35 UTC, monarch_dodra wrote:
1. Performance (small issue). A lot of code that should be
otherwise fast is slower than it could be, since druntime will
check, at runtime, if postblit is needed, if destruction should be
done. For example, if you "dup" a string by hand, you can do it
50% faster than what druntime does.

Yes, this is pitfall of D design.

I don't see what this has to do with D design. It's just a quality of implementation issue, isn't it?
2. Inference. This is an odd one. Currently, the function that
operate on arrays (reserve, dup, length) etc... may or may not be
 safe or pure. Or nothrow. The problem is that we really can't do
anything about it. duping a string is obviously nothrow, but if
you dup something with a postblit, then it might not be (nor
safe).

Yes, this is pitfall of D design.

It's just a matter of taking care of the details. Dup'ing an array involves allocating a new array (pure safe nothrow) and copying elements over (purity/safety/nothrowness depends on postblit). So there should be two versions of dup, reserve, etc., one for arrays whose elements can be copied in a pure/safe/nothrow way, one for arrays where the safety/etc. of copying elements depends on safety/etc. of the element postblits. The latter can be taken care of with attribute inference. So for example, we could do something like this: T[] dup(T)(T[] array) pure safe nothrow if (!hasElaborateCopyConstructor!(T[])) { // To reduce template bloat, forward to static function // that does bitwise copying. return simpleDupImpl(array.ptr, array.size, T.sizeof); } T[] dup(T)(T[] array) /* attributes inferred at compile-time */ if (hasElaborateCopyConstructor!(T[])) { // This part is pure safe nothrow T[] copy = simpleDupImpl(array.ptr, array.size, T.sizeof); // This part's purity/safety/nothrowness inherits from // postblit's characteristics. callPostblits!T(copy); }
These issues kind of keep popping up in phobos, and we will have
to address them sooner or later. The current state of affairs is a
kind of status quo of "its wrong, inconsistent, but maybe OK most
of the time".


Yeah this is one thing that irks me about D. It works superbly -- for the most part. But when you hit something outside "the most part", then it's a cascade of troubles and roadblocks, one leading to the other. We really need to dig into all these details and straighten them out, otherwise it's going to be a big source of frustration to potential D adopters.
 They are systematically popping up because of systematical language
 features (D is statically typed language with little runtime
 supported).

And how do you propose we address that?
For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.


Why is assumeSafeAppend not nothrow? If it were up to me, I'd say that if for whatever reason safe append can't be done (i.e. the assumption is invalid) and we detect it, we should assert(0), since the code is obviously not prepared to deal with this case by virtue of calling assumeSafeAppend in the first place. Throwing an Exception makes no sense in this case -- it's not something you can handle.
========

The solution (I think) would be to partially templatize d-runtime.
We can keep most of what is in rt/lifetime.d, but statically split
it into smaller chunks, which could be piloted with more accuracy
via a templated "manager".

It depends on how do you want to templatize it. Due to separate compilation model it is impossible to know beforehand (when you provide phobos library together with druntime) which types will be used.

Right, so we use templates for the "front-end" that interface directly with user types, but the real work is done by static backend functions. There will be a different backend function to deal with each different category of cases, and the front-end templates will just forward to the most appropriate one.
For now, if we could make a change as simple as "no postblit =>
safe pure nothrow, and fast guaranteed"/"postblit => not safe, not
pure, may throw", then it would already be a big win (IMO).

========

I had started toying around with this, and the main and immediate
challenge I ran into is the integration of templates. The 2 issues
are:
1. All templates must be in object.d, or the compiler won't see
it.


public import is your friend. :)
2. A simple (and needed) trait like "hasElaborateCopyConstructor"
requires a good third of both "typetuple.d" and "traits.d".
"functionAttributes" is pretty simple though (I think).

Yes, this is pitfall of D design.

How is this a problem with D design? I'd argue that a large part of std.traits *should* be in druntime, not Phobos, because they interface with compiler internals via __traits. After "discovering" how parameter tuples work yesterday, I'm becoming more convinced that __traits should not be used by user code at all, but wrapped by more user-friendly functions in the runtime environment, i.e., druntime. One of the issues I ran into while trying to reimplement the built-in AA's was that it needed some stuff from std.traits, but can't use it because druntime isn't supposed to depend on Phobos. The duplication in that case was relatively small, but if std.traits was in druntime in the first place, this would never have been an issue.
I'm mostly looking to start a discussion on this, and for
guidance, ideas, on where to start.

I doubt that the problems you raised are solvable in general case with current runtime capabilities. While it still may be possible to tackle some of them, D design (separate compilation model, type system, templates, how pure, safe, nothrow are defined) would prevent you from fixing these problems.

The upcoming 2.064 has attribute inference for pure/ safe/nothrow. I'm pretty sure this will address most, if not all, of these issues, if applied correctly. T -- Laissez-faire is a French term commonly interpreted by Conservatives to mean 'lazy fairy,' which is the belief that if governments are lazy enough, the Good Fairy will come down from heaven and do all their work for them.
Sep 13 2013
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 13 September 2013 at 17:02:24 UTC, H. S. Teoh wrote:
For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.


Why is assumeSafeAppend not nothrow? If it were up to me, I'd say that if for whatever reason safe append can't be done (i.e. the assumption is invalid) and we detect it, we should assert(0), since the code is obviously not prepared to deal with this case by virtue of calling assumeSafeAppend in the first place. Throwing an Exception makes no sense in this case -- it's not something you can handle.

Well, by your own conclusion, it doesn't throw :D It either just works, or it errors out. No exceptions.
I had started toying around with this, and the main and 
immediate
challenge I ran into is the integration of templates. The 2 
issues
are:
1. All templates must be in object.d, or the compiler won't 
see
it.


public import is your friend. :)

What I'm getting so far from this conversation, is we should move select "foundation blocks" of typetuple/traits from phobos, into druntime. the phobos libraries would then publicly import their druntime counterparts. Then object.d would import "core/traits.d" and "core/typetuple.d" (or whatever). Why "public" import btw? Wouldn't that expose the functions in traits/typetuple to *all* D code? That seems like a bad idea... One of the problems though is that we'd want to try to keep those "core/traits" libraries as light as possible, since they'd be pretty much imported 99% of the time. Another issue to solve is that the above would work well for say, "reserve". However, "dup" (afaik) is hardwired by the compiler to call the druntime function "_adDupT". SO we'd have to change that if we even *hope* for it to infer anything.
Sep 13 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Sep 13, 2013 at 07:47:32PM +0200, monarch_dodra wrote:
 On Friday, 13 September 2013 at 17:02:24 UTC, H. S. Teoh wrote:
For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.


Why is assumeSafeAppend not nothrow? If it were up to me, I'd say that if for whatever reason safe append can't be done (i.e. the assumption is invalid) and we detect it, we should assert(0), since the code is obviously not prepared to deal with this case by virtue of calling assumeSafeAppend in the first place. Throwing an Exception makes no sense in this case -- it's not something you can handle.

Well, by your own conclusion, it doesn't throw :D It either just works, or it errors out. No exceptions.

Right, so how come assumeSafeAppend isn't marked nothrow?
I had started toying around with this, and the main and immediate
challenge I ran into is the integration of templates. The 2 issues
are: 1. All templates must be in object.d, or the compiler won't
see it.


public import is your friend. :)

What I'm getting so far from this conversation, is we should move select "foundation blocks" of typetuple/traits from phobos, into druntime. the phobos libraries would then publicly import their druntime counterparts.

Either that, or factor std.traits in terms of a small number of fundamental primitives, and move those primitives into druntime.
 Then object.d would import "core/traits.d" and "core/typetuple.d"
 (or whatever).
 
 Why "public" import btw? Wouldn't that expose the functions in
 traits/typetuple to *all* D code? That seems like a bad idea...

This was in response to your question about why stuff must be in object.d(i) otherwise it won't be seen. If it's only needed internally, then a public import is not necessary, but if you want stuff to be imported by default into all programs, but don't want to bloat object.di, then public import is the solution. I've thought before, about splitting up object_.d into more manageable chunks. Public imports would allow us to do this in a transparent, backwards-compatible way.
 One of the problems though is that we'd want to try to keep those
 "core/traits" libraries as light as possible, since they'd be pretty
 much imported 99% of the time.

Yes, that's why I said above that we should distill the stuff we need from std.traits into their barest essentials, and put those in druntime. Then std.traits itself can add more user conveniences around these primitives.
 Another issue to solve is that the above would work well for say,
 "reserve". However, "dup" (afaik) is hardwired by the compiler to
 call the druntime function "_adDupT". SO we'd have to change that if
 we even *hope* for it to infer anything.

Oh joy, more compiler magic. Let me know when you figure out how to extricate it from the compiler, since it will probably be useful in extricating AA's from the compiler as well. ;) OTOH, another approach, since the compiler already hardcodes this stuff, is to make the *compiler* infer attributes for .dup. That might actually be easier than trying to move compiler-hardcoded stuff into druntime. (The downside is that you'll have to work with C++... :-P) T -- Klein bottle for rent ... inquire within. -- Stephen Mulraney
Sep 13 2013
prev sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 13 September 2013 at 18:01:30 UTC, H. S. Teoh wrote:
 On Fri, Sep 13, 2013 at 07:47:32PM +0200, monarch_dodra wrote:
 On Friday, 13 September 2013 at 17:02:24 UTC, H. S. Teoh wrote:
For example, "dup" is not nothrow, yet it is safe and pure.
Reserve is nothrow, yet assumeSafeAppend is not. Reserve may
actually call postblit, but not assumeSafeAppend.


Why is assumeSafeAppend not nothrow? If it were up to me, I'd say that if for whatever reason safe append can't be done (i.e. the assumption is invalid) and we detect it, we should assert(0), since the code is obviously not prepared to deal with this case by virtue of calling assumeSafeAppend in the first place. Throwing an Exception makes no sense in this case -- it's not something you can handle.

Well, by your own conclusion, it doesn't throw :D It either just works, or it errors out. No exceptions.

Right, so how come assumeSafeAppend isn't marked nothrow?

Oh. Right. I understood you backwards. In that case, it's not nothrow simply because it hasn't been made that way yet. On topic: https://github.com/D-Programming-Language/druntime/pull/553 - Me trying to make assumeSafeAppend nothrow, and learn how druntime works.
 Another issue to solve is that the above would work well for 
 say,
 "reserve". However, "dup" (afaik) is hardwired by the compiler 
 to
 call the druntime function "_adDupT". SO we'd have to change 
 that if
 we even *hope* for it to infer anything.

Oh joy, more compiler magic. Let me know when you figure out how to extricate it from the compiler, since it will probably be useful in extricating AA's from the compiler as well. ;) OTOH, another approach, since the compiler already hardcodes this stuff, is to make the *compiler* infer attributes for .dup. That might actually be easier than trying to move compiler-hardcoded stuff into druntime. (The downside is that you'll have to work with C++... :-P) T

Hum. I wouldn't mind "starting" with reserve, and see from there what the best thing to do for dup/idup is from there (including just making it a free function in object ?) An added plus about making it an actual UFCS in object.d, is that we'll be able to (correctly) write it as "myArray.dup()". It's not a property god damn it!
Sep 13 2013