www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - api: One attribute to rule them All

reply "Zach the Mystic" <reachzach gggmail.com> writes:
Hello everybody. My name is Zach, and I have a suggestion for the 
improvement of D. I've been looking at the following stalled pull 
request for a while now:

https://github.com/D-Programming-Language/dmd/pull/1877

...in which Walter Bright wants to introduce built-in-attribute 
inference for a relatively small set of functions. It seems like 
the most obvious thing in the world to me to desire this, and not 
even just for 'auto' and templated functions, but for *every* 
function. And there's no reason it can't be done. So long as the 
compiler has everything it needs to determine which attributes 
can be applied, there's no reason to demand anything from the 
programmer. Look how simple this function is:

int plusOne(int a) { return a+1; }

Let's say I later want to call it, however, from a fully 
attributed function:

int plusTwo(int a) pure nothrow  safe  nogc  {
   return plusOne(plusOne(a));
}

I get a compiler error. The only way to stop it is to add 
unnecessary visual noise to the first function. All of these 
attributes should be something that you *want* to add, not 
something that you *need*. The compiler can obviously figure out 
if the function throws or not. Just keep an additional internal 
flag for each of the attributes. When any attribute is violated, 
flip the bit and boom, you have your implicit function signature.

I think this is how it always should have been. It's important to 
remember that the above attributes have the 'covariant' property, 
which means they can always be called by any function without 
that property. Therefore no existing code will start failing to 
compile. Only certain things which would have *errored* before 
will stop. Plus new optimizations can be done.

So what's the problem? As you can read in the vehement opposition 
to pull 1877 above, the big fear is that function signatures will 
start changing willy-nilly, causing the exposed interface of the 
function to destabilize, which will cause linker errors or 
require code intended to be kept separate in large projects to be 
recompiled at every little change.

I find this depressing! That something so good should be ruined 
by something so remote as the need for separate compilation in 
very large projects? I mean, most projects aren't even very 
large. Also, because D compiles so much faster than its 
predecessors, is it even such a big deal to have to recompile 
everything?

But let's admit the point may be valid. Yes, under attribute 
inference, the function signatures in the exposed API will indeed 
find themselves changing every time one so much as adds a 
'printf' or calls something that throws.

But they don't *have* to change. The compiler doesn't need to 
include the inferred attributes when it generates the mangled 
name and the .di signature, only the explicit ones. From within 
the program, all the opportunities for inference and optimization 
could be left intact, while outside programs accessing the code 
in precompiled form could only access the functions as explicitly 
indicated.

This makes no change to the language, except that it allows new 
things to compile. The only hitch is this: What if you want the 
full advantages of optimization and inference from across 
compilation boundaries? You'd have to add each of the covariant 
function attributes manually to every function you exposed. From 
my perspective, this is still a chore.

I suggest a new attribute,  api, which does nothing more than to 
tell the compiler to generate the function signature and mangle 
the name only with its explicit attributes, and not with its 
inferred ones. Inside the program, there's no reason the compiler 
can't continue to use inference, but with  api, the exposed 
interface will be stabilized, should the programmer want that. 
Simple.

I anticipate a couple of objections to my proposal:

The first is that we would now demand that the programmer decide 
whether he wants his exposed functions stabilized or not. For a 
large library used by different people, this choice might pose 
some difficulty. But it's not that bad. You just choose: do you 
want to improve compilation times and/or closed-source 
consistency by ensuring a stable interface, or do you want to 
speed up runtime performance without having to clutter your code? 
Most projects would choose the latter.  api is made available for 
the those who don't. The opposition to attribute inference put 
forth in pull 1877 is thereby appeased.

A second objection to this proposal: Another attribute? Really? 
Well, yeah.

But it's not a problem, I say, for these reasons:

1. This one little attribute allows you to excise gajillions of 
unnecessary little attributes which are currently forced on the 
programmer by the lack of inference, simply by appeasing the 
opponents of inference and allowing it to be implemented.

2. It seems like most people will be okay just recompiling 
projects instead of preferring to stabilize their apis. Thus, 
 api will only be used rarely.

3.  api forces you to add all the attributes you want exposed to 
the world manually. It's a candid admission that you are okay 
littering your code with attributes, thereby lessening the pain 
at having to add one more.

4. Most  api functions will come in clusters. After all, it *is* 
an API you are exposing, so I think it's highly likely that a 
single " api:" will work in most cases.


Now, "Bombard with your gunships."

Thank you.
Jan 05 2015
next sibling parent reply "Daniel N" <ufo orbiting.us> writes:
On Monday, 5 January 2015 at 21:15:00 UTC, Zach the Mystic wrote:

 Now, "Bombard with your gunships."


An alternative could be to use the already existing 'export'.

http://dlang.org/attribute.html
"Export means that any code outside the executable can access the 
member. Export is analogous to exporting definitions from a DLL."
Jan 05 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 5 January 2015 at 21:25:01 UTC, Daniel N wrote:

 An alternative could be to use the already existing 'export'.


'extern'. Yeah, something like 'extern (noinfer):'.
Jan 05 2015
parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 5 January 2015 at 22:00:40 UTC, Zach the Mystic wrote:

 On Monday, 5 January 2015 at 21:25:01 UTC, Daniel N wrote:

 An alternative could be to use the already existing 'export'.


 'extern'. Yeah, something like 'extern (noinfer):'.


Err, yeah, whatever works!
Jan 05 2015
prev sibling next sibling parent reply Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 05/01/15 22:14, Zach the Mystic via Digitalmars-d wrote:

 I get a compiler error. The only way to stop it is to add unnecessary visual
 noise to the first function. All of these attributes should be something that
 you *want* to add, not something that you *need*. The compiler can obviously
 figure out if the function throws or not. Just keep an additional internal flag
 for each of the attributes. When any attribute is violated, flip the bit and
 boom, you have your implicit function signature.


Bear in mind one quite important factor -- all that alleged noise isn't simply 
about getting stuff to work, it's about promises that the function makes to 
downstream users.  You do touch on this yourself, but I think you have missed 
how your  api flag could go wrong.


 I suggest a new attribute,  api, which does nothing more than to tell the
 compiler to generate the function signature and mangle the name only with its
 explicit attributes, and not with its inferred ones. Inside the program,
there's
 no reason the compiler can't continue to use inference, but with  api, the
 exposed interface will be stabilized, should the programmer want that. Simple.


IMHO if anything like this is to be implemented, the extra flag should be to 
indicate that a function is _not_ intended to be part of the API and that 
therefore it is OK to infer its attributes.

Here's the rationale.  Suppose that I have a bunch of functions that are all 
intended to be part of the public API of my project.  I accidentally forget to 
tag one of them with the  api attribute, so its attributes will be 
auto-inferred, but the function is still public, so downstream users will wind 
up using it.

3 months later, I realize my mistake, and add the  api attribute -- at which 
point downstream users' code will break if their code was relying on the 
unintended inferred attributes.

If on the other hand you take the assumption that attributes should by default 
_not_ be auto-inferred, and you accidentally forget to tag a function to 
auto-infer its attributes, that can be fixed without breaking downstream.

It's quite analogous in this respect to the argument about final vs. virtual by 
default for class methods.
Jan 05 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 5 January 2015 at 23:48:17 UTC, Joseph Rushton 
Wakeling via Digitalmars-d wrote:

 Here's the rationale.  Suppose that I have a bunch of functions 
 that are all intended to be part of the public API of my 
 project.  I accidentally forget to tag one of them with the 
  api attribute,


A more likely scenario is that your library starts small enough 
not to need the  api attribute, then at some point it gets 
really, really huge. Then in one fell swoop you decide to " api:" 
your whole file so that the public interface won't change so 
often. I'm picking the most extreme case I can think of, in order 
to argue the point from a different perspective.


 so its attributes will be auto-inferred, but the function is 
 still public, so downstream users will wind up using it.

 3 months later, I realize my mistake, and add the  api 
 attribute -- at which point downstream users' code will break 
 if their code was relying on the unintended inferred attributes.


Attribute inference provides convenience, not guarantees. If a 
user was relying on the purity of a function which was never 
marked 'pure', it's only convenience which allows him to do it, 
both on the part of the user, for adding 'pure', and the library 
writer, for *not* adding it. Adding  api (or 'extern (noinfer)') 
cancels that convenience for the sake of modularity. It's a 
tradeoff. The problem  itself is solved either by the library 
writer marking the function 'pure', or the user removing 'pure' 
from his own function. Without  api, the problem only arises when 
the library writer actually does something impure, which makes 
perfect sense. It's  api (and D's existing default, by the way) 
which adds the artificiality to the process, not my suggested 
default.


 It's quite analogous in this respect to the argument about 
 final vs. virtual by default for class methods.


I don't think so, because of so-called covariance. Final and 
virtual each have their own advantages and disadvantages, whereas 
inferring attributes only goes one way. There is no cost to 
inferring in the general case. My suggestion, (I now prefer 
'extern(noinfer)'), does absolutely nothing except to restore D's 
existing default, for what I think are the rare cases it is 
needed. I could be wrong about just how rare using 
extern(noinfer) will actually be, but consider that phobos, for 
example, just doesn't need it, because it's too small a library 
to cause trouble if all of a sudden one of its non-templated 
functions becomes impure. A quick recompile, a new interface 
file, and now everyone's using the new thing. Even today, it's 
not even marked up with attributes completely, thus indicating 
that you never even *could* have used it for all it's worth.

Have I convinced you?
Jan 05 2015
parent reply "Joseph Rushton Wakeling" <joseph.wakeling webdrake.net> writes:
On Tuesday, 6 January 2015 at 03:29:39 UTC, Zach the Mystic wrote:

 A more likely scenario is that your library starts small enough 
 not to need the  api attribute, then at some point it gets 
 really, really huge. Then in one fell swoop you decide to 
 " api:" your whole file so that the public interface won't 
 change so often. I'm picking the most extreme case I can think 
 of, in order to argue the point from a different perspective.


Note that if you want auto-inferred attributes during the alpha 
phase of library development, it's just as trivial to put a 
general  autoinfer: or  noapi: or whatever you like, and that in 
turn is a pretty nice signifier to the user "this function's 
attributes are not guaranteed to be stable".


 Attribute inference provides convenience, not guarantees.


Indeed.  But any publicly-available API is a guarantee of sorts.  
 From the moment people are using something, you can no longer 
vicariously break things.


 If a user was relying on the purity of a function which was 
 never marked 'pure', it's only convenience which allows him to 
 do it, both on the part of the user, for adding 'pure', and the 
 library writer, for *not* adding it.


Nevertheless, if a user relies on that inferred purity (which 
they will do), and you tweak things so the function is no longer 
pure, you have broken that downstream user's code.  Worse, you'll 
have done it in a way which flies under the radar until someone 
actually tries to build against your updated library.  You, as 
the library developer, won't get any automatic warning that 
you've broken backwards compatibility with your earlier 
implementation; the downstream user won't get any 
automatically-documented warnings of this breaking change.

If instead you have an explicit "please auto-infer attributes for 
this function" marker, then at least the user has a very clear 
warning that any attributes possessed by this function cannot be 
relied on.  (Absence of a guarantee != presence of a statement 
that there is NO guarantee:-)


 Adding  api (or 'extern (noinfer)') cancels that convenience 
 for the sake of modularity. It's a tradeoff. The problem  
 itself is solved either by the library writer marking the 
 function 'pure', or the user removing 'pure' from his own 
 function.


As a library writer, I don't think you can responsibly expect 
users to bear the burden of fixing undocumented breaking change.


 Without  api,  the problem only arises when the library writer
 actually does something impure, which makes perfect sense.
 It's  api (and D's existing default, by the way) which adds
 the artificiality to the process, not my suggested default.


I'm not sure what exactly you mean when you talk about D's 
existing default, but one aspect that I think is important is: 
D's default position is that a function has no guarantees, and 
you _add_ guarantees to it via attributes.

This whole discussion would be quite different if the default was 
that a function is expected to be  safe, pure, nothrow, etc., and 
the developer is expected to use attributes to indicate 
_weakening_ of those guarantees.


 It's quite analogous in this respect to the argument about 
 final vs. virtual by default for class methods.


 I don't think so, because of so-called covariance. Final and 
 virtual each have their own advantages and disadvantages, 
 whereas inferring attributes only goes one way. There is no 
 cost to inferring in the general case.


I think you have missed the point I was making.

If you have final-by-default for classes, and you accidentally 
forget to tag a public method as 'virtual', then you can fix that 
without breaking any downstream user's code.  If by contrast you 
have virtual-by-default and you accidentally forget to tag a 
public method as 'final', then you can't fix that without the 
risk of breaking downstream; _someone_ may have relied on that 
function being virtual.

The situation is very similar here.  If your function has no 
attributes, and then later you add one (say, 'pure'), then you 
don't do any downstream user any harm.  If on the other hand your 
function _does_ have attributes -- whether explicit or inferred 
-- and then you remove them, you risk breaking downstream code.

If you don't auto-infer, this is not really an issue, because you 
have to manually add and remove attributes, and so you can never 
unintentionally or unknowingly remove an attribute.  But if you 
_do_ auto-infer, then it's very easy indeed to accidentally 
remove attributes that your downstream may have relied on.


 My suggestion, (I now prefer 'extern(noinfer)'), does absolutely
 nothing except to restore D's existing default, for what I think
 are the rare cases it is needed. I could be wrong about just how
 rare using extern(noinfer) will actually be, but consider that
 phobos, for example, just doesn't need it, because it's too 
 small
 a library to cause trouble if all of a sudden one of its
 non-templated functions becomes impure.


I don't think you can reasonably anticipate how much trouble 
breaking change can cause for your downstreams.  At a minimum, 
_accidental_ and undocumented breaking change is completely 
unacceptable, and this proposal introduces many easy ways to see 
it happen.


 A quick recompile, a new interface file, and now everyone's 
 using the new thing. Even today, it's not even marked up with 
 attributes completely, thus indicating that you never even 
 *could* have used it for all it's worth.

 Have I convinced you?


I understand why you want it, I just think you underestimate the 
importance of avoiding accidentally breaking things for 
downstream.

Note that I'd be much more prepared to be convinced if the 
proposal was that attributes should be auto-inferred _except_ for 
public functions, although that has its own range of nastinesses 
in terms of inconsistent behaviour.
Jan 06 2015
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 6 January 2015 at 12:07:21 UTC, Joseph Rushton 
Wakeling wrote:

 As a library writer, I don't think you can responsibly expect 
 users to bear the burden of fixing undocumented breaking change.


I agree, maybe just replace "module" with "library". Also make 
"module" mandatory. It takes no extra keyword if it always is the 
first token.

Then the compiler can be more lax with "module" files and more 
strict with "library" files.
Jan 06 2015
prev sibling next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 6 January 2015 at 12:07:21 UTC, Joseph Rushton 
Wakeling wrote:

 If you have final-by-default for classes, and you accidentally 
 forget to tag a public method as 'virtual', then you can fix 
 that without breaking any downstream user's code.  If by 
 contrast you have virtual-by-default and you accidentally 
 forget to tag a public method as 'final', then you can't fix 
 that without the risk of breaking downstream; _someone_ may 
 have relied on that function being virtual.


That's a much bigger problem than what you get with inferred 
attributes. If you have whole class hierarchy built around a 
function which isn't even supposed to be virtual, you'll have a 
lot of refactoring to do when it's marked final.

The only way in which downstream code will ever "break" is when 
the called function is longer expressed by the interface as 
'pure'. Under the new default, this will only happen when the 
function *actually stops being pure*. This is good! The only code 
which will then break is code which is itself marked 'pure'. The 
solution is either to remove 'pure' from the user's function, or 
to stop calling the impure function. It's not the end of the 
world if a user function stops being pure ( safe, nothrow, etc.). 
Since the called function was never marked 'pure', and thus never 
guaranteed to be pure to begin with, it was only luck and 
convenience which allowed the user to tag his code 'pure' in the 
first place. And, as John Colvin just pointed out, with the new 
defaults, the user most likely won't even bother to tag his code 
'pure', because he knows he will get all the optimization 
benefits without the hassle. The only reason to tag 'pure' from 
now on will be to *guarantee* it. At that point, you *want* your 
code to break, but only when your function actually stops being 
pure.

With D's existing default (and with code marked with the 
suggested new attribute 'extern(noinfer)'), you have another 
reason why your 'pure' function won't compile. Indeed, it's 
already broken. I ask you, why should a user have to wait until 
someone marks a function 'pure' to get the advantages of its 
purity, when the compiler already has everything it needs in 
order to grant those advantages? In the rare case where you 
actually want to revert to the way D is now, then you simply have 
to pay the price, which is the *same* price that everyone is 
already paying now.
Jan 06 2015
prev sibling parent "Elie Morisse" <syniurge gmail.com> writes:
On Tuesday, 6 January 2015 at 12:07:21 UTC, Joseph Rushton 
Wakeling wrote:

 I think you have missed the point I was making.

 If you have final-by-default for classes, and you accidentally 
 forget to tag a public method as 'virtual', then you can fix 
 that without breaking any downstream user's code.  If by 
 contrast you have virtual-by-default and you accidentally 
 forget to tag a public method as 'final', then you can't fix 
 that without the risk of breaking downstream; _someone_ may 
 have relied on that function being virtual.



For people making libraries which really need to keep ABI 
compatibility there should be some quick trick to tell the 
compiler to build those ABI-stable interfaces like:

   class {
   final  api:
     ...

   virtual  api:
     ...
   }

It's imho a small price to pay for not having to always write 
explicit attributes for most D functions.
Jan 06 2015
prev sibling next sibling parent Joseph Rushton Wakeling via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 06/01/15 00:48, Joseph Rushton Wakeling via Digitalmars-d wrote:

 IMHO if anything like this is to be implemented, the extra flag should be to
 indicate that a function is _not_ intended to be part of the API and that
 therefore it is OK to infer its attributes.


Hmm.  On thinking about this some more, it occurs to me that this might be 
fundamentally about protection.  If it were forbidden to auto-infer attributes 
for a non-templated public function, then quite a few of my objections above 
might go away.
Jan 05 2015
prev sibling next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 05 Jan 2015 21:14:58 +0000
Zach the Mystic via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 Hello everybody. My name is Zach, and I have a suggestion for the=20
 improvement of D. I've been looking at the following stalled pull=20
 request for a while now:
=20
 https://github.com/D-Programming-Language/dmd/pull/1877


heh. i did a little hack based on this patch: autoinference is turned
on only for `private auto`. i also added a bunch of UDAs to control the
process: ` inferattr` (can be applied to any function), ` notinferattr`,
` canthrow`, ` impure` and ` gc` (to control inference).

any explicit attribute on function will block inference too.

as druntime and phobos has no `private auto` which is not templated or
without explicit attribues, it compiles fine. and for my code i have
some control.

this feature can be poorly designed, but as it doesn't conflict with
most of the existing code, i'm happy with it. one should be carefull
with templates calling private functions with inferred attributes (it
breaks linking -- for obvious reason), but it's ok for me. `private
auto` is so ugly that it will rise my alarm level anyway. i mean
"function returning 'auto' is suspicious".
Jan 06 2015
prev sibling next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 5 January 2015 at 21:15:00 UTC, Zach the Mystic wrote:

 Hello everybody. My name is Zach, and I have a suggestion for 
 the improvement of D. I've been looking at the following 
 stalled pull request for a while now:

 https://github.com/D-Programming-Language/dmd/pull/1877

 ...in which Walter Bright wants to introduce built-in-attribute 
 inference for a relatively small set of functions. It seems 
 like the most obvious thing in the world to me to desire this, 
 and not even just for 'auto' and templated functions, but for 
 *every* function. And there's no reason it can't be done. So 
 long as the compiler has everything it needs to determine which 
 attributes can be applied, there's no reason to demand anything 
 from the programmer. Look how simple this function is:

 int plusOne(int a) { return a+1; }

 Let's say I later want to call it, however, from a fully 
 attributed function:

 int plusTwo(int a) pure nothrow  safe  nogc  {
   return plusOne(plusOne(a));
 }

 I get a compiler error. The only way to stop it is to add 
 unnecessary visual noise to the first function. All of these 
 attributes should be something that you *want* to add, not 
 something that you *need*. The compiler can obviously figure 
 out if the function throws or not. Just keep an additional 
 internal flag for each of the attributes. When any attribute is 
 violated, flip the bit and boom, you have your implicit 
 function signature.

 I think this is how it always should have been. It's important 
 to remember that the above attributes have the 'covariant' 
 property, which means they can always be called by any function 
 without that property. Therefore no existing code will start 
 failing to compile. Only certain things which would have 
 *errored* before will stop. Plus new optimizations can be done.

 So what's the problem? As you can read in the vehement 
 opposition to pull 1877 above, the big fear is that function 
 signatures will start changing willy-nilly, causing the exposed 
 interface of the function to destabilize, which will cause 
 linker errors or require code intended to be kept separate in 
 large projects to be recompiled at every little change.

 I find this depressing! That something so good should be ruined 
 by something so remote as the need for separate compilation in 
 very large projects? I mean, most projects aren't even very 
 large. Also, because D compiles so much faster than its 
 predecessors, is it even such a big deal to have to recompile 
 everything?

 But let's admit the point may be valid. Yes, under attribute 
 inference, the function signatures in the exposed API will 
 indeed find themselves changing every time one so much as adds 
 a 'printf' or calls something that throws.

 But they don't *have* to change. The compiler doesn't need to 
 include the inferred attributes when it generates the mangled 
 name and the .di signature, only the explicit ones. From within 
 the program, all the opportunities for inference and 
 optimization could be left intact, while outside programs 
 accessing the code in precompiled form could only access the 
 functions as explicitly indicated.

 This makes no change to the language, except that it allows new 
 things to compile. The only hitch is this: What if you want the 
 full advantages of optimization and inference from across 
 compilation boundaries? You'd have to add each of the covariant 
 function attributes manually to every function you exposed. 
 From my perspective, this is still a chore.

 I suggest a new attribute,  api, which does nothing more than 
 to tell the compiler to generate the function signature and 
 mangle the name only with its explicit attributes, and not with 
 its inferred ones. Inside the program, there's no reason the 
 compiler can't continue to use inference, but with  api, the 
 exposed interface will be stabilized, should the programmer 
 want that. Simple.

 I anticipate a couple of objections to my proposal:

 The first is that we would now demand that the programmer 
 decide whether he wants his exposed functions stabilized or 
 not. For a large library used by different people, this choice 
 might pose some difficulty. But it's not that bad. You just 
 choose: do you want to improve compilation times and/or 
 closed-source consistency by ensuring a stable interface, or do 
 you want to speed up runtime performance without having to 
 clutter your code? Most projects would choose the latter.  api 
 is made available for the those who don't. The opposition to 
 attribute inference put forth in pull 1877 is thereby appeased.

 A second objection to this proposal: Another attribute? Really? 
 Well, yeah.

 But it's not a problem, I say, for these reasons:

 1. This one little attribute allows you to excise gajillions of 
 unnecessary little attributes which are currently forced on the 
 programmer by the lack of inference, simply by appeasing the 
 opponents of inference and allowing it to be implemented.

 2. It seems like most people will be okay just recompiling 
 projects instead of preferring to stabilize their apis. Thus, 
  api will only be used rarely.

 3.  api forces you to add all the attributes you want exposed 
 to the world manually. It's a candid admission that you are 
 okay littering your code with attributes, thereby lessening the 
 pain at having to add one more.

 4. Most  api functions will come in clusters. After all, it 
 *is* an API you are exposing, so I think it's highly likely 
 that a single " api:" will work in most cases.


 Now, "Bombard with your gunships."

 Thank you.


Needing a function to have a completely stable signature is the 
(very important) exception, not the rule and inference is awesome 
for everywhere else.

It's worth noting that with full inference then much less code 
would be explicitly annotated, which goes some way towards 
ameliorating the stability problem as more client code will be 
flexible w.r.t. changing attributes on library boundaries.

About restriction to `auto`:
`auto` is about return types, not attributes. Inference should be 
performed on anything that's available. *.di files should contain 
the inferred attributes when generated to support totally 
separate compilation. For more convoluted examples, tooling could 
help (compiler spits out json with inference results, external 
tool / IDE picks up that info and helps the user automatically 
apply it to declarations elsewhere).
Jan 06 2015
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 6 January 2015 at 09:12:43 UTC, John Colvin wrote:

 On Monday, 5 January 2015 at 21:15:00 UTC, Zach the Mystic 
 wrote:

 Hello everybody. My name is Zach, and I have a suggestion for 
 the improvement of D. I've been looking at the following 
 stalled pull request for a while now:

 https://github.com/D-Programming-Language/dmd/pull/1877

 ...in which Walter Bright wants to introduce 
 built-in-attribute inference for a relatively small set of 
 functions. It seems like the most obvious thing in the world 
 to me to desire this, and not even just for 'auto' and 
 templated functions, but for *every* function. And there's no 
 reason it can't be done. So long as the compiler has 
 everything it needs to determine which attributes can be 
 applied, there's no reason to demand anything from the 
 programmer. Look how simple this function is:

 int plusOne(int a) { return a+1; }

 Let's say I later want to call it, however, from a fully 
 attributed function:

 int plusTwo(int a) pure nothrow  safe  nogc  {
  return plusOne(plusOne(a));
 }

 I get a compiler error. The only way to stop it is to add 
 unnecessary visual noise to the first function. All of these 
 attributes should be something that you *want* to add, not 
 something that you *need*. The compiler can obviously figure 
 out if the function throws or not. Just keep an additional 
 internal flag for each of the attributes. When any attribute 
 is violated, flip the bit and boom, you have your implicit 
 function signature.

 I think this is how it always should have been. It's important 
 to remember that the above attributes have the 'covariant' 
 property, which means they can always be called by any 
 function without that property. Therefore no existing code 
 will start failing to compile. Only certain things which would 
 have *errored* before will stop. Plus new optimizations can be 
 done.

 So what's the problem? As you can read in the vehement 
 opposition to pull 1877 above, the big fear is that function 
 signatures will start changing willy-nilly, causing the 
 exposed interface of the function to destabilize, which will 
 cause linker errors or require code intended to be kept 
 separate in large projects to be recompiled at every little 
 change.

 I find this depressing! That something so good should be 
 ruined by something so remote as the need for separate 
 compilation in very large projects? I mean, most projects 
 aren't even very large. Also, because D compiles so much 
 faster than its predecessors, is it even such a big deal to 
 have to recompile everything?

 But let's admit the point may be valid. Yes, under attribute 
 inference, the function signatures in the exposed API will 
 indeed find themselves changing every time one so much as adds 
 a 'printf' or calls something that throws.

 But they don't *have* to change. The compiler doesn't need to 
 include the inferred attributes when it generates the mangled 
 name and the .di signature, only the explicit ones. From 
 within the program, all the opportunities for inference and 
 optimization could be left intact, while outside programs 
 accessing the code in precompiled form could only access the 
 functions as explicitly indicated.

 This makes no change to the language, except that it allows 
 new things to compile. The only hitch is this: What if you 
 want the full advantages of optimization and inference from 
 across compilation boundaries? You'd have to add each of the 
 covariant function attributes manually to every function you 
 exposed. From my perspective, this is still a chore.

 I suggest a new attribute,  api, which does nothing more than 
 to tell the compiler to generate the function signature and 
 mangle the name only with its explicit attributes, and not 
 with its inferred ones. Inside the program, there's no reason 
 the compiler can't continue to use inference, but with  api, 
 the exposed interface will be stabilized, should the 
 programmer want that. Simple.

 I anticipate a couple of objections to my proposal:

 The first is that we would now demand that the programmer 
 decide whether he wants his exposed functions stabilized or 
 not. For a large library used by different people, this choice 
 might pose some difficulty. But it's not that bad. You just 
 choose: do you want to improve compilation times and/or 
 closed-source consistency by ensuring a stable interface, or 
 do you want to speed up runtime performance without having to 
 clutter your code? Most projects would choose the latter.  api 
 is made available for the those who don't. The opposition to 
 attribute inference put forth in pull 1877 is thereby appeased.

 A second objection to this proposal: Another attribute? 
 Really? Well, yeah.

 But it's not a problem, I say, for these reasons:

 1. This one little attribute allows you to excise gajillions 
 of unnecessary little attributes which are currently forced on 
 the programmer by the lack of inference, simply by appeasing 
 the opponents of inference and allowing it to be implemented.

 2. It seems like most people will be okay just recompiling 
 projects instead of preferring to stabilize their apis. Thus, 
  api will only be used rarely.

 3.  api forces you to add all the attributes you want exposed 
 to the world manually. It's a candid admission that you are 
 okay littering your code with attributes, thereby lessening 
 the pain at having to add one more.

 4. Most  api functions will come in clusters. After all, it 
 *is* an API you are exposing, so I think it's highly likely 
 that a single " api:" will work in most cases.


 Now, "Bombard with your gunships."

 Thank you.


 Needing a function to have a completely stable signature is the 
 (very important) exception, not the rule and inference is 
 awesome for everywhere else.

 It's worth noting that with full inference then much less code 
 would be explicitly annotated, which goes some way towards 
 ameliorating the stability problem as more client code will be 
 flexible w.r.t. changing attributes on library boundaries.

 About restriction to `auto`:
 `auto` is about return types, not attributes. Inference should 
 be performed on anything that's available. *.di files should 
 contain the inferred attributes when generated to support 
 totally separate compilation. For more convoluted examples, 
 tooling could help (compiler spits out json with inference 
 results, external tool / IDE picks up that info and helps the 
 user automatically apply it to declarations elsewhere).


tldr: I like what you're thinking, please can we have this.
Jan 06 2015
next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
 tldr: I like what you're thinking, please can we have this.


+1

Atila
Jan 06 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 6 January 2015 at 09:14:00 UTC, John Colvin wrote:

 tldr: I like what you're thinking, please can we have this.


Hey thanks, John.
Jan 06 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Tuesday, 6 January 2015 at 09:12:43 UTC, John Colvin wrote:

 It's worth noting that with full inference then much less code 
 would be explicitly annotated, which goes some way towards 
 ameliorating the stability problem as more client code will be 
 flexible w.r.t. changing attributes on library boundaries.


That's a great point. Explicit attributes will only be for larger 
projects which really want those guarantees. Everyone else gets 
all the optimizations for free... pretty cool!
Jan 06 2015
prev sibling next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
Martin Nowak is the one who created the issue for this 
enhancement request:

https://issues.dlang.org/show_bug.cgi?id=10979

He proposes a different solution though - creating two mangled 
names for each function. The advantage, as far as I can see it, 
is that you wouldn't even have to add " api" or "extern(noinfer)" 
to link properly. I see two disadvantages: the binary would 
suffer bloat, and the generated .di signatures would have to 
default to the conservative explicit signature, since is could 
not include both signatures without causing a duplicate 
definition.
Jan 07 2015
parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Thursday, 8 January 2015 at 04:43:50 UTC, Zach the Mystic 
wrote:

 https://issues.dlang.org/show_bug.cgi?id=10979


Whoops! The link is:
https://issues.dlang.org/show_bug.cgi?id=10924
Jan 07 2015
prev sibling next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 5 January 2015 at 21:15:00 UTC, Zach the Mystic wrote:

 https://github.com/D-Programming-Language/dmd/pull/1877


Deep into the discussion on the above pull, Walter responds to 
this comment of Kenji's:

"I think this kind of optimization should be hidden after the 
semantic phase - as like inlining and backend optimizations."

...with this:

"The trouble with doing this is that it is extremely dangerous in 
a language that supports separate compilation. It means that a 
function can be treated as pure, but its mangled signature says 
its impure. Then, the build system swaps in another version of 
that function, with the same signature, but is impure. Now the 
code that calls it is silently broken - the worst kind of runtime 
error.
This is why attribute inference is limited to those functions 
where the definitions are guaranteed to be available - templates, 
lambdas, nesteds and autos - then there can be no runtime 
mismatch."

What bothers me about this is that the compiler could *know* that 
a function is pure any time it has the function body available to 
it, regardless of what kind of function it is. The above quote 
suggests, however, that it's just completely impossible for the 
compiler to know if the function which will be called is the one 
that it has just compiled, or if it will be swapped in by a 
linker with another function with the same name from somewhere 
else. I honestly don't know enough about how linking works to 
know if this cancels my proposal. Wouldn't it cause an error if 
the linker found two of the exact same function?

My naive assumption about it is that when the compiler has the 
body, it can treat it as pure from within, even if it gets 
expressed by 'extern(noinfer)' as impure. That is, from within 
the singly compiled program, it can be treated as if it *is* 
pure, being one-hundred percent certain that the function it 
compiles will be the one that gets called at binary time, but for 
anyone reading the expressed interface, it can only be compiled 
as if it's *not* pure, since that's all the information the 
interface tells them.
Jan 07 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
I think that push for more inference / WPO is an important goal 
for D. However I had somewhat more radical and generic changes in 
mind, ones that don't add new keywords or semantics but rather 
strictly define what existing ones mean. This was supposed to be 
turned into a DIP at some point (possibly after consulting with 
Walter) but here is the draft outline:

- separate compilation in basic C sense becomes illegal
- minimal legal unit of separate compilation is defined as static 
library
- any time library gets built, it _must_ be distributed with 
matching .di interfaces. If there are original .d files imported, 
one must not try to link prebuilt library.
- .di generation is split in two modes:
     1) 'minimal' (API) which only writes exported symbols and 
ignores even public ones. All inferred attributes gets written 
explicitly there. This is what gets recommended for public 
distribution (even if it is source-only distribution) and what 
defines stable API.
     2) 'full' mode which is similar to existing .di generation 
but with all attributes explicitly written to all functions. It 
is approach recommended for creating intermediate built artifacts 
(such as dub building of dependencies).

Stability of (1) headers can be validated mechanically by 
compiler / external tool in such scenario. As you may notice no 
new keywords / language concepts are proposed, it is only about 
more strict definition of standard development flow. It also 
opens well-defined borderline for any WPO.

Needs a lot of work before any serious destruction of course but 
should give some overall feeling of what I am going at.
Jan 09 2015
next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Friday, 9 January 2015 at 11:40:28 UTC, Dicebot wrote:

 I think that push for more inference / WPO is an important goal 
 for D. However I had somewhat more radical and generic changes 
 in mind, ones that don't add new keywords or semantics but 
 rather strictly define what existing ones mean. This was 
 supposed to be turned into a DIP at some point (possibly after 
 consulting with Walter) but here is the draft outline:

 - separate compilation in basic C sense becomes illegal
 - minimal legal unit of separate compilation is defined as 
 static library
 - any time library gets built, it _must_ be distributed with 
 matching .di interfaces. If there are original .d files 
 imported, one must not try to link prebuilt library.


This last one is where all the threats from bad linking seem to 
come from.


 - .di generation is split in two modes:
     1) 'minimal' (API) which only writes exported symbols and 
 ignores even public ones. All inferred attributes gets written 
 explicitly there. This is what gets recommended for public 
 distribution (even if it is source-only distribution) and what 
 defines stable API.
     2) 'full' mode which is similar to existing .di generation 
 but with all attributes explicitly written to all functions. It 
 is approach recommended for creating intermediate built 
 artifacts (such as dub building of dependencies).


To sum up: All .di function signatures will now be generated with 
both explicit and inferred attributes. The keyword 'export' will 
be overloaded with a new meaning, toggled on and off by a 
compiler flag which generates .di files based on that meaning. 
Correct?


 Stability of (1) headers can be validated mechanically by 
 compiler / external tool in such scenario. As you may notice no 
 new keywords / language concepts are proposed, it is only about 
 more strict definition of standard development flow. It also 
 opens well-defined borderline for any WPO.


If solving the problem at the level of the command line with the 
help of the existing 'export' attribute is more flexible and 
robust, then I'm all for it. The first thing to find out is if 
anyone will have a problem overloading the meaning of 'export' 
for this purpose. I can't think of a reason they would, unless 
people are currently using 'export' in some niche way which would 
be ruined by the new flag.


 Needs a lot of work before any serious destruction of course 
 but should give some overall feeling of what I am going at.


We both agree that full "covariant" inference should be the 
default. It seems like the first step is to introduce it using an 
opt-in compiler flag (e.g. "-infer"). I'm not sure how .di files 
and mangled names should be generated using this flag though. The 
goal is to get the kinks out of the inference process before 
making it the default. What's step two, though?
Jan 09 2015
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 9 January 2015 at 16:14:01 UTC, Zach the Mystic wrote:

 If solving the problem at the level of the command line with 
 the help of the existing 'export' attribute is more flexible 
 and robust, then I'm all for it. The first thing to find out is 
 if anyone will have a problem overloading the meaning of 
 'export' for this purpose. I can't think of a reason they 
 would, unless people are currently using 'export' in some niche 
 way which would be ruined by the new flag.


This DIP is of relevance:
http://wiki.dlang.org/DIP45
Jan 10 2015
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 10 January 2015 at 10:47:04 UTC, Marc Schütz wrote:

 On Friday, 9 January 2015 at 16:14:01 UTC, Zach the Mystic 
 wrote:

 If solving the problem at the level of the command line with 
 the help of the existing 'export' attribute is more flexible 
 and robust, then I'm all for it. The first thing to find out 
 is if anyone will have a problem overloading the meaning of 
 'export' for this purpose. I can't think of a reason they 
 would, unless people are currently using 'export' in some 
 niche way which would be ruined by the new flag.


 This DIP is of relevance:
 http://wiki.dlang.org/DIP45


Yes, it was partly what started me thinking in this direction, 
DIP45 is very wise.
Jan 11 2015
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 9 January 2015 at 16:14:01 UTC, Zach the Mystic wrote:

 To sum up: All .di function signatures will now be generated 
 with both explicit and inferred attributes. The keyword 
 'export' will be overloaded with a new meaning, toggled on and 
 off by a compiler flag which generates .di files based on that 
 meaning. Correct?


It is not truly overloading with new meaning. One can argue that 
existing meaning of export can already be interpreted that way, 
it is simply not implemented in full power. See DIP45 for some 
more stuff on topic.

As for the rest - I'd better try answering those questions if I 
ever sit down and actually write it carefully, right now it is 
more like bunch of vague ideas rolling around with no solid 
structure.
Jan 11 2015
prev sibling parent "Atila Neves" <atila.neves gmail.com> writes:
Very interesting, looking forward to reading the DIP.

Atila

On Friday, 9 January 2015 at 11:40:28 UTC, Dicebot wrote:

 I think that push for more inference / WPO is an important goal 
 for D. However I had somewhat more radical and generic changes 
 in mind, ones that don't add new keywords or semantics but 
 rather strictly define what existing ones mean. This was 
 supposed to be turned into a DIP at some point (possibly after 
 consulting with Walter) but here is the draft outline:

 - separate compilation in basic C sense becomes illegal
 - minimal legal unit of separate compilation is defined as 
 static library
 - any time library gets built, it _must_ be distributed with 
 matching .di interfaces. If there are original .d files 
 imported, one must not try to link prebuilt library.
 - .di generation is split in two modes:
     1) 'minimal' (API) which only writes exported symbols and 
 ignores even public ones. All inferred attributes gets written 
 explicitly there. This is what gets recommended for public 
 distribution (even if it is source-only distribution) and what 
 defines stable API.
     2) 'full' mode which is similar to existing .di generation 
 but with all attributes explicitly written to all functions. It 
 is approach recommended for creating intermediate built 
 artifacts (such as dub building of dependencies).

 Stability of (1) headers can be validated mechanically by 
 compiler / external tool in such scenario. As you may notice no 
 new keywords / language concepts are proposed, it is only about 
 more strict definition of standard development flow. It also 
 opens well-defined borderline for any WPO.

 Needs a lot of work before any serious destruction of course 
 but should give some overall feeling of what I am going at.
Jan 10 2015
prev sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
I've now created DIP70 for this issue:

http://wiki.dlang.org/DIP70

Maybe a more sophisticated linking mechanism will make DIP70 
unnecessary, but it should fuel the discussion nonetheless. Also, 
the wiki refused all of my attempts to insert the following 
links, saying they were non-canonical, despite my following the 
instructions on the help page:

LINK1 (this thread):
http://forum.dlang.org/thread/vlzwhhymkjgckgyoxlrq forum.dlang.org

LINK2 (Dicebot's proposal):
http://forum.dlang.org/post/otejdbgnhmyvbyaxatsk forum.dlang.org

Please edit the wiki to correct this if you can.
Jan 17 2015