www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Things that make writing a clean binding system more difficult

reply Ethan Watson <gooberman gmail.com> writes:
As mentioned in the D blog the other day, the binding system as 
used by Remedy will both be open sourced and effectively 
completely rewritten from when we shipped Quantum Break. As I'm 
still deep within that rewrite, a bunch of things are still fresh 
in my mind that aren't that great when it comes to D and doing 
such a system.

These are things I also expect other programmers to come across 
in one way or another, being that they seem like a simple way to 
do things but getting them to behave require non-trivial 
workarounds.

I also assume "lodge a bug" will be the response to these. But 
there are some cases where I think documentation or 
easily-googleable articles will be required instead/as well. And 
in the case of one of these things, it's liable to start a long 
circular conversation chain.

====================
1) Declaring a function pointer with a ref return value can't be 
done without workarounds.

Try compiling this:

ref int function( int, int ) functionPointer;

It won't let you, because only parameters and for loop symbols 
can be ref types. Despite the fact that I intend the function 
pointer to be of a kind that returns a ref int, I can't declare 
that easily. Easy, declare an alias, right?

alias RefFunctionPointer = ref int function( int, int );

Alright, cool, that works. But thanks to the binding system 
making heavy use of function pointers via code-time generated 
code, that means we then have to come up with a unique name for 
every function pointer symbol we'll need. Eep.

Rather, I have to do something like this:

template RefFunctionPointer( Params... ) if( Params.length > 1 )
{
   ref Params[ 0 ] dodgyFunction( Params[ 1 .. $ ] );
   alias RefFunctionPointer = typeof( &dodgyFunction );
}
RefFunctionPointer!( int, int, int ) functionPointer;

This can also alternately be done by generating a mixin string 
for the alias inside of the template and not requiring a dummy 
function to get the type from. Either way, it gets rid of the 
unique name requirement but now we have template expansion in the 
mix. Which is something I'll get to in a second...

Needless to say, this is something I wasted a lot of time on 
three years ago when I was getting the bindings up to speed 
originally. Turns out it's not any better in DMD 2.071.

====================
2) Expansion of code (static foreach, templates) is slow to the 
point where string mixins are a legitimate compile-time 
optimisation

Take an example of whittling down a tuple/variable argument list. 
Doing it recursively would look something like this:

template SomeEliminator( Symbols... )
{
   static if( Symbols.length >= 1 )
   {
     static if( SomeCondition!( Symbol[ 0 ] ) )
     {
       alias SomeEliminator = TypeTuple!( Symbol[ 0 ], Symbols[ 1 
.. $ ] );
     }
     else
     {
       alias SomeEliminator = TypeTuple!( Symbols[ 1 .. $ ] );
     }
   }
   else
   {
     alias SomeEliminator = TypeTuple!( );
   }
}

Okay, that works, but the template expansion is a killer on 
compile-time performance. It's legitimately far quicker on the 
compiler to do this:

template SomeEliminator( Symbols... )
{
   string SymbolSelector()
   {
     string[] strOutputs;
     foreach( iIndex, Symbol; Symbols )
     {
       static if( SomeCondition!( Symbol ) )
       {
         strOutputs ~= "Symbols[ " ~ iIndex.stringof ~ " ]";
       }
     }
     return strOutputs.joinWith( ", " );
   }
   mixin( "alias SomeEliminator = TypeTuple!( " ~ SymbolSelector() 
~ " );" );
}

With just a small codebase that I'm working on here, it chops 
seconds off the compile time. Of course, maybe there's something 
I'm missing here about variable parameter parsing and doing it 
without a mixin is quite possible and just as quick as the mixin, 
but that would make it the third method I know of to achieve the 
same effect. The idiomatic way of doing this without mixins 
should at least be defined, and optimised at the compiler level 
so that people don't get punished for writing natural D code.

Then there was this one that I came across:

outofswitch: switch( symbolName )
{
   foreach( Variable; VariablesOf!( SearchType ) )
   {
     case Variable.Name:
       doSomething!( Variable.Type )();
       break outofswitch;
   }
   default:
     writeln( symbolName, " was not found!" );
     break;
}

This caused compile time to blow way out. How far out? By 
rewriting it like this, I cut compile times in half (at that 
point, from 10 seconds to 5):

switch( symbolName )
{
   mixin( generateSwitchFor!( SearchType )() );
   default:
     writeln( symbolName, " was not found!" );
     break;
}

Now, I love mixins, both template form and string form. The 
binding system uses them extensively. But mixins like this are 
effectively a hack. Anytime I have to break out a mixin because 
my compile time doubled from a seemingly simple piece of code is 
not good.

====================
3) __ctfe is not a CTFE symbol.

This one bit me when I was trying to be efficient for runtime 
usage while allowing a function to also be usable at compile time.

int[] someArray;
static if( !__ctfe )
{
   someArray.reserve( someAmount );
}

Reserve not working in compile time? Eh, I can live with that. 
__ctfe not being a symbol I can static if with? Well, sure, I 
suppose that would work if the compiler wouldn't even try 
compiling the code inside the __ctfe block. But it doesn't do 
that. It does symbol resolution, and then your code doesn't run 
at compile time. It's at that point where I ask why have the 
__ctfe symbol if you can only use it effectively at runtime? 
Doesn't that only make it half useful?

I understand this is a longstanding complaint too, so this serves 
as a reminder.

====================
4) Forward declaring a function prototype means I can never 
declare that function elsewhere (say, for example, with a mixin)

The binding system works something like this:

* Declare a function, mark it with a  BindImport UDA.
* Compile time code scans over objects and symbols looking for 
functions tagged  BindImport.
* Generate __gshared function pointers that match the signature 
(and rewrite parameters to pass this in where applicable).
* Generate function definitions that call the function pointers 
(with this if it's a method), allowing a programmer to just call 
the function declaration like it was any old ordinary piece of D 
code.

That fourth part is where it all falls over.

We shipped Quantum Break by defining your imports with a 
preceding underscore (ie  BindImport int _doSomeAction();) and 
generated function definitions with the exact same signature 
minus the underscore. The new way I'm doing it is to define all 
these functions in a sub-struct so that all I need to rewrite is 
the parameters.

All this because I cannot later define a forward-declared 
function.

.di files are both not a language feature (documentation notes it 
is explicitly a compiler feature), and don't even match what I 
need here as they're generated from complete code with no 
possibility of using them in more of a .cpp/.h paradigm. So 
they're out of the question. *Unless* they're upgraded to a 
language feature and allow me to define full class declarations 
with later implementations of some/all functions.

This also isn't the only use case I have. I'm a game engine 
programmer. We write a lot of abstracted interfaces with platform 
specific implementations. I know, I know, version(X){} your code, 
right? But that's not how everyone works. Some implementations 
really do require their own file for maintenance and legal 
purposes. But for an example outside of gaming? Take a look at 
core.atomic. Two implementations in the same file *AND* a 
separate one for documentation purposes. LDC's core.atomic also 
has an LLVM definition in there. And if someone writes native ARM 
support for DMD, that'll be even more implementations in the same 
file. Take note of the duplicated enums and deprecations between 
definitions, the alternative to which is to put a version block 
inside every function that requires special behaviour. Either way 
is not clean, I tells ya.

I'm sure there's many cases where declaration and later 
definition is also a perfectly valid programming pattern, and I 
don't see at all how these use cases can conflict with D's 
forward referencing since it doesn't change referencing rules at 
all, it only changes the definition rules.
Jul 28 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/28/2016 1:33 AM, Ethan Watson wrote:

 1) Declaring a function pointer with a ref return value can't be done without
 workarounds.

 Try compiling this:

 ref int function( int, int ) functionPointer;

 It won't let you, because only parameters and for loop symbols can be ref
types.
 Despite the fact that I intend the function pointer to be of a kind that
returns
 a ref int, I can't declare that easily. Easy, declare an alias, right?

 alias RefFunctionPointer = ref int function( int, int );


C/C++ have essentially the same problem, if you want to declare a function 
pointer parameter that has different linkage.

The trouble is there's an ambiguity in the grammar. I don't really have
anything 
better than the two step process you outlined.



 ====================
 4) Forward declaring a function prototype means I can never declare that
 function elsewhere (say, for example, with a mixin)


Do you mean:

   void foo();
   void foo() { }

?
Jul 28 2016
next sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Thursday, 28 July 2016 at 08:49:35 UTC, Walter Bright wrote:

 Do you mean:

   void foo();
   void foo() { }

 ?


Exactly this. I've been unable to get it to work.
Jul 28 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/28/2016 1:54 AM, Ethan Watson wrote:

 On Thursday, 28 July 2016 at 08:49:35 UTC, Walter Bright wrote:

 Do you mean:

   void foo();
   void foo() { }

 ?


 Exactly this. I've been unable to get it to work.


https://issues.dlang.org/show_bug.cgi?id=16329

The reason it's an enhancement request rather than a bug is that with D's 
support for forward referenced functions, having to declare them first followed 
later by a definition was deemed unnecessary.

This pattern is, of course, necessary in C/C++ because they do not allow
forward 
referenced declarations outside of aggregates.
Jul 28 2016
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thursday, July 28, 2016 01:49:35 Walter Bright via Digitalmars-d wrote:

 On 7/28/2016 1:33 AM, Ethan Watson wrote:

 1) Declaring a function pointer with a ref return value can't be done
 without workarounds.

 Try compiling this:

 ref int function( int, int ) functionPointer;

 It won't let you, because only parameters and for loop symbols can be ref
 types. Despite the fact that I intend the function pointer to be of a
 kind that returns a ref int, I can't declare that easily. Easy, declare
 an alias, right?

 alias RefFunctionPointer = ref int function( int, int );


 C/C++ have essentially the same problem, if you want to declare a function
 pointer parameter that has different linkage.

 The trouble is there's an ambiguity in the grammar. I don't really have
 anything better than the two step process you outlined.


Well, if we decided to make parens with ref legal, then we could make it
work. e.g.

ref(int) function(int, int) functionPointer;

Now, I don't know of any other case where you'd actually use parens with ref
if it were legal, but it would solve this particular case if we wanted to
provide a way around the ambiguity.

- Jonathan M Davis
Jul 28 2016
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 28 July 2016 at 20:16:11 UTC, Jonathan M Davis wrote:

 Well, if we decided to make parens with ref legal, then we 
 could make it work. e.g.

 ref(int) function(int, int) functionPointer;

 Now, I don't know of any other case where you'd actually use 
 parens with ref if it were legal, but it would solve this 
 particular case if we wanted to provide a way around the 
 ambiguity.

 - Jonathan M Davis


On a somewhat related tangent, I was looking for the history of 
why ref was included in the language. My recollection of Andrei's 
book is that it just takes ref as a given, instead of pass by 
reference or address in C++, rather than say why that decision 
was made. I found that ref was added in D 1.011 as a replacement 
for inout. Looking through some of the D 1.0 documentation, I see 
that
"C++ does not distinguish between in, out and ref (i.e. inout) 
parameters."
but not much else.
Jul 28 2016
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 7/28/16 4:16 PM, Jonathan M Davis via Digitalmars-d wrote:

 On Thursday, July 28, 2016 01:49:35 Walter Bright via Digitalmars-d wrote:

 On 7/28/2016 1:33 AM, Ethan Watson wrote:

 1) Declaring a function pointer with a ref return value can't be done
 without workarounds.

 Try compiling this:

 ref int function( int, int ) functionPointer;

 It won't let you, because only parameters and for loop symbols can be ref
 types. Despite the fact that I intend the function pointer to be of a
 kind that returns a ref int, I can't declare that easily. Easy, declare
 an alias, right?

 alias RefFunctionPointer = ref int function( int, int );


 C/C++ have essentially the same problem, if you want to declare a function
 pointer parameter that has different linkage.

 The trouble is there's an ambiguity in the grammar. I don't really have
 anything better than the two step process you outlined.


 Well, if we decided to make parens with ref legal, then we could make it
 work. e.g.

 ref(int) function(int, int) functionPointer;

 Now, I don't know of any other case where you'd actually use parens with ref
 if it were legal, but it would solve this particular case if we wanted to
 provide a way around the ambiguity.


No, because that implies a type-modifier. ref does not modify the type 
at all, it just specifies the storage class.

-Steve
Jul 28 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 29 July 2016 at 07:34, Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 7/28/16 4:16 PM, Jonathan M Davis via Digitalmars-d wrote:

 On Thursday, July 28, 2016 01:49:35 Walter Bright via Digitalmars-d wrote:

 On 7/28/2016 1:33 AM, Ethan Watson wrote:

 1) Declaring a function pointer with a ref return value can't be done
 without workarounds.

 Try compiling this:

 ref int function( int, int ) functionPointer;

 It won't let you, because only parameters and for loop symbols can be
 ref
 types. Despite the fact that I intend the function pointer to be of a
 kind that returns a ref int, I can't declare that easily. Easy, declare
 an alias, right?

 alias RefFunctionPointer = ref int function( int, int );



 C/C++ have essentially the same problem, if you want to declare a
 function
 pointer parameter that has different linkage.

 The trouble is there's an ambiguity in the grammar. I don't really have
 anything better than the two step process you outlined.



 Well, if we decided to make parens with ref legal, then we could make it
 work. e.g.

 ref(int) function(int, int) functionPointer;

 Now, I don't know of any other case where you'd actually use parens with
 ref
 if it were legal, but it would solve this particular case if we wanted to
 provide a way around the ambiguity.



 No, because that implies a type-modifier. ref does not modify the type at
 all, it just specifies the storage class.


Ah yes, one of the most fundamental and terrible mistake in D ;)
Aug 04 2016
prev sibling parent Kagamin <spam here.lot> writes:
On Thursday, 28 July 2016 at 20:16:11 UTC, Jonathan M Davis wrote:

 Well, if we decided to make parens with ref legal, then we 
 could make it work. e.g.

 ref(int) function(int, int) functionPointer;

 Now, I don't know of any other case where you'd actually use 
 parens with ref if it were legal, but it would solve this 
 particular case if we wanted to provide a way around the 
 ambiguity.


I had an idea of putting function attributes between return type 
and function name:
int ref function(int, int)
Jul 29 2016
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 28.07.2016 10:49, Walter Bright wrote:

 On 7/28/2016 1:33 AM, Ethan Watson wrote:

 1) Declaring a function pointer with a ref return value can't be done
 without
 workarounds.

 Try compiling this:

 ref int function( int, int ) functionPointer;

 It won't let you, because only parameters and for loop symbols can be
 ref types.
 Despite the fact that I intend the function pointer to be of a kind
 that returns
 a ref int, I can't declare that easily. Easy, declare an alias, right?

 alias RefFunctionPointer = ref int function( int, int );


 C/C++ have essentially the same problem, if you want to declare a
 function pointer parameter that has different linkage.

 The trouble is there's an ambiguity in the grammar. I don't really have
 anything better than the two step process you outlined.
 ...


My parser accepts the following:

int function(int,int)ref functionPointer;

I wasn't really aware that this was illegal in DMD. (Other function 
attributes, such as pure, are accepted.)

In fact, even the following is disallowed:
int foo(int)ref{}


Should I file an enhancement request?
Jul 28 2016
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, July 29, 2016 06:44:16 Timon Gehr via Digitalmars-d wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other function
 attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


Except that ref isn't a function attribute. It's an attribute on the return
type. So, it doesn't make sense for it to be on the right. That would be
like having the const on the right-hand side of a member function apply to
the return type rather than the function itself.

- Jonathan M Davis
Jul 28 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 29.07.2016 06:52, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 06:44:16 Timon Gehr via Digitalmars-d wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other function
 attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


 Except that ref isn't a function attribute.


Yes it is.

int x;
ref{
     int foo(){ return x;}
}
pragma(msg, typeof(&foo()));



 It's an attribute on the return type.


There is no such thing. Types cannot have attributes.


 So, it doesn't make sense for it to be on the right. That would be
 like having the const on the right-hand side of a member function apply to
 the return type rather than the function itself.
 ...


You have it backwards.
Jul 28 2016
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, July 29, 2016 08:29:19 Timon Gehr via Digitalmars-d wrote:

 On 29.07.2016 06:52, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 06:44:16 Timon Gehr via Digitalmars-d wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other function
 attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


 Except that ref isn't a function attribute.


 Yes it is.

 int x;
 ref{
      int foo(){ return x;}
 }
 pragma(msg, typeof(&foo()));


That's downright bizzarre given that ref applies to the return type and not
to the this pointer (and that function doesn't even have a this pointer,
since it's not a member function).


 It's an attribute on the return type.


 There is no such thing. Types cannot have attributes.


Sure they can. e.g.

auto func(ref int param) {...}

The same with in and out. They apply to the type of the parameter without
actually being part of it.


 So, it doesn't make sense for it to be on the right. That would be
 like having the const on the right-hand side of a member function apply to
 the return type rather than the function itself.
 ...


 You have it backwards.


It looks to me like the compiler is treating ref in a schizophrenic manner
given that when it's used on a parameter, it treats it as part of the
parameter, whereas with the return type, it's treating it as a function
attribute instead of associating it with the return type. I'd guess that
that stems from the fact that while ref is really supposed to be associated
with the type, it's not actually part of the type.

- Jonathan M Davis
Jul 28 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 29.07.2016 08:51, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 08:29:19 Timon Gehr via Digitalmars-d wrote:

 On 29.07.2016 06:52, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 06:44:16 Timon Gehr via Digitalmars-d wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other function
 attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


 Except that ref isn't a function attribute.


 Yes it is.

 int x;
 ref{
      int foo(){ return x;}
 }
 pragma(msg, typeof(&foo()));


 That's downright bizzarre given that ref applies to the return type and not
 to the this pointer (and that function doesn't even have a this pointer,
 since it's not a member function).
 ...


It does not apply to the return type. There is actually no way to tell 
whether a function returns ref or not by just examining the return type. 
It's the function which is ref.


 It's an attribute on the return type.


 There is no such thing. Types cannot have attributes.


 Sure they can. e.g.

 auto func(ref int param) {...}

 The same with in and out. They apply to the type of the parameter without
 actually being part of it.
 ...


No, typeof(param) is int. The type is not changed at all. The attribute 
applies to the parameter declaration.


 So, it doesn't make sense for it to be on the right. That would be
 like having the const on the right-hand side of a member function apply to
 the return type rather than the function itself.
 ...


 You have it backwards.


 It looks to me like the compiler is treating ref in a schizophrenic manner
 given that when it's used on a parameter, it treats it as part of the
 parameter, whereas with the return type, it's treating it as a function
 attribute instead of associating it with the return type. I'd guess that
 that stems from the fact that while ref is really supposed to be associated
 with the type, it's not actually part of the type.

 - Jonathan M Davis


'ref' has nothing to do with the type. This is not C++.

The only thing that is inconsistent here is that 'ref' is not accepted 
on the right for function declarations.
Jul 29 2016
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, July 29, 2016 09:03:18 Timon Gehr via Digitalmars-d wrote:

 'ref' has nothing to do with the type. This is not C++.

 The only thing that is inconsistent here is that 'ref' is not accepted
 on the right for function declarations.


ref may not be part of the type, but it just seems totally wrong for it to
be applying to anything else. I mean, how does it make any sense for a
_function_ to be ref or not? It may return by ref, but the function itself
is a function. It's what's going on with the return value that changes based
or ref or not.

Maybe I'm just looking at this wrong, but it seems completely bizarre to
consider the _function_ to be ref or not. It sounds like it's just a weird
biproduct of trying to make it so that ref doesn't propagate beyond the
return type or the parameter so that you can't have ref variables in
general.

I've always looked at D's ref as being essentially the same as C++'s &
except that it's not considered to be part of the type, just attached to it
in a way that doesn't propagate. The same with with in or out. I just don't
see how it even makes conceptual sense for ref to be an attribute of the
function itself.

- Jonathan M Davis
Jul 29 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/29/2016 1:34 AM, Jonathan M Davis via Digitalmars-d wrote:

 I've always looked at D's ref as being essentially the same as C++'s &
 except that it's not considered to be part of the type, just attached to it
 in a way that doesn't propagate. The same with with in or out. I just don't
 see how it even makes conceptual sense for ref to be an attribute of the
 function itself.


C++'s & is a bizarre type constructor in that it has completely wacky and 
special cased behavior in just about everything involving types, including type 
deduction, type inference, overloading, etc. For example, you can't have a 
pointer to a ref. Or a ref of a ref. Sometimes the ref is considered part of
the 
type, sometimes not. Etc.

With D, making it a sort of storage class completely sidesteps that mess.
Jul 29 2016
next sibling parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, July 29, 2016 02:55:14 Walter Bright via Digitalmars-d wrote:

 On 7/29/2016 1:34 AM, Jonathan M Davis via Digitalmars-d wrote:

 I've always looked at D's ref as being essentially the same as C++'s &
 except that it's not considered to be part of the type, just attached to
 it
 in a way that doesn't propagate. The same with with in or out. I just
 don't
 see how it even makes conceptual sense for ref to be an attribute of the
 function itself.


 C++'s & is a bizarre type constructor in that it has completely wacky and
 special cased behavior in just about everything involving types, including
 type deduction, type inference, overloading, etc. For example, you can't
 have a pointer to a ref. Or a ref of a ref. Sometimes the ref is considered
 part of the type, sometimes not. Etc.

 With D, making it a sort of storage class completely sidesteps that mess.


I understand that part. It's treating it like a function attribute that
makes no sense to me. Even if it's not treated as part of the return type
exactly, it's still the return type that it's affecting, not the function.
ref, in, and out are all in this kind of weird place where they affect a
type without being part of the type, and I'm guessing that because of how
oddball they are, there really wasn't a good way to deal with ref on the
return type, since it wasn't actually part of the type, so in the
implementation, it got tied to the function instead rather than trying to
add a new concept to the compiler. So, I guess that from an implementation
perspective, it makes some sense, but it's still downright weird given that
ref really has to do with the return type and not the function itself, even
if ref is a storage class rather than actually being part of the type.

- Jonathan M Davis
Jul 29 2016
parent reply Dicebot <public dicebot.lv> writes:
On 07/29/2016 02:05 PM, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 02:55:14 Walter Bright via Digitalmars-d wrote:

 On 7/29/2016 1:34 AM, Jonathan M Davis via Digitalmars-d wrote:

 I've always looked at D's ref as being essentially the same as C++'s &
 except that it's not considered to be part of the type, just attached to
 it
 in a way that doesn't propagate. The same with with in or out. I just
 don't
 see how it even makes conceptual sense for ref to be an attribute of the
 function itself.


 C++'s & is a bizarre type constructor in that it has completely wacky and
 special cased behavior in just about everything involving types, including
 type deduction, type inference, overloading, etc. For example, you can't
 have a pointer to a ref. Or a ref of a ref. Sometimes the ref is considered
 part of the type, sometimes not. Etc.

 With D, making it a sort of storage class completely sidesteps that mess.


 
 I understand that part. It's treating it like a function attribute that
 makes no sense to me. Even if it's not treated as part of the return type
 exactly, it's still the return type that it's affecting, not the function.
 ref, in, and out are all in this kind of weird place where they affect a
 type without being part of the type, and I'm guessing that because of how
 oddball they are, there really wasn't a good way to deal with ref on the
 return type, since it wasn't actually part of the type, so in the
 implementation, it got tied to the function instead rather than trying to
 add a new concept to the compiler. So, I guess that from an implementation
 perspective, it makes some sense, but it's still downright weird given that
 ref really has to do with the return type and not the function itself, even
 if ref is a storage class rather than actually being part of the type.


What you want it contradictory to the concept of "storage class".
Jul 29 2016
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, July 29, 2016 14:14:49 Dicebot via Digitalmars-d wrote:

 On 07/29/2016 02:05 PM, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 02:55:14 Walter Bright via Digitalmars-d wrote:

 On 7/29/2016 1:34 AM, Jonathan M Davis via Digitalmars-d wrote:

 I've always looked at D's ref as being essentially the same as C++'s &
 except that it's not considered to be part of the type, just attached to
 it
 in a way that doesn't propagate. The same with with in or out. I just
 don't
 see how it even makes conceptual sense for ref to be an attribute of the
 function itself.


 C++'s & is a bizarre type constructor in that it has completely wacky and
 special cased behavior in just about everything involving types,
 including
 type deduction, type inference, overloading, etc. For example, you can't
 have a pointer to a ref. Or a ref of a ref. Sometimes the ref is
 considered
 part of the type, sometimes not. Etc.

 With D, making it a sort of storage class completely sidesteps that mess.


 I understand that part. It's treating it like a function attribute that
 makes no sense to me. Even if it's not treated as part of the return type
 exactly, it's still the return type that it's affecting, not the function.
 ref, in, and out are all in this kind of weird place where they affect a
 type without being part of the type, and I'm guessing that because of how
 oddball they are, there really wasn't a good way to deal with ref on the
 return type, since it wasn't actually part of the type, so in the
 implementation, it got tied to the function instead rather than trying to
 add a new concept to the compiler. So, I guess that from an implementation
 perspective, it makes some sense, but it's still downright weird given
 that
 ref really has to do with the return type and not the function itself,
 even
 if ref is a storage class rather than actually being part of the type.


 What you want it contradictory to the concept of "storage class".


Why? I thought the the whole idea of "storage class" was that it was an
attribute that was applied to a type that wasn't actually part of the type.
Certainly, based on past discussions on the topic, it seems to be that the
term is used for pretty much anything that gets applied to a type that isn't
actually part of the type. And that's definitely what happens with ref.

- Jonathan M Davis
Jul 29 2016
parent Dicebot <public dicebot.lv> writes:
On 07/29/2016 03:55 PM, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 14:14:49 Dicebot via Digitalmars-d wrote:

 What you want it contradictory to the concept of "storage class".


 
 Why? I thought the the whole idea of "storage class" was that it was an
 attribute that was applied to a type that wasn't actually part of the type.
 Certainly, based on past discussions on the topic, it seems to be that the
 term is used for pretty much anything that gets applied to a type that isn't
 actually part of the type. And that's definitely what happens with ref.
 
 - Jonathan M Davis


Storage class has to attach to some symbol to modify its semantics
without affecting the type, it doesn't apply to type at all. Usually it
is a variable and that comes naturally but in case of function return
value / parameters the only available symbol is function itself -
contrary to variables, parameters are not available as independent
symbols AFAIK.

That said, `ref` isn't a 100% storage class, otherwise `alias Foo = ref
int function ( )` wouldn't work. Judging by only observable semantics
I'd say it is more akin to type qualifier contrary to what spec says -
but qualifier of function type.
Jul 29 2016
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 29 July 2016 at 19:55, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 7/29/2016 1:34 AM, Jonathan M Davis via Digitalmars-d wrote:

 I've always looked at D's ref as being essentially the same as C++'s &
 except that it's not considered to be part of the type, just attached to
 it
 in a way that doesn't propagate. The same with with in or out. I just
 don't
 see how it even makes conceptual sense for ref to be an attribute of the
 function itself.



 C++'s & is a bizarre type constructor in that it has completely wacky and
 special cased behavior in just about everything involving types, including
 type deduction, type inference, overloading, etc. For example, you can't
 have a pointer to a ref. Or a ref of a ref. Sometimes the ref is considered
 part of the type, sometimes not. Etc.

 With D, making it a sort of storage class completely sidesteps that mess.


But introduces a MUCH bigger mess in its place. D's ref is absolute
torture past the most simple of cases. The design doesn't scale, it's
one gigantic uncontrollable edge case built into the core of the
language, with no tools to wrangle or manage. I've never had such an
agonizing time with C++'s & as I have with ref in D, and I've had a
LOT of agonizing time with C++ ;)
Aug 04 2016
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 29 July 2016 at 16:51, Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Friday, July 29, 2016 08:29:19 Timon Gehr via Digitalmars-d wrote:

 On 29.07.2016 06:52, Jonathan M Davis via Digitalmars-d wrote:

 On Friday, July 29, 2016 06:44:16 Timon Gehr via Digitalmars-d wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other function
 attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


 Except that ref isn't a function attribute.


 Yes it is.

 int x;
 ref{
      int foo(){ return x;}
 }
 pragma(msg, typeof(&foo()));


 That's downright bizzarre given that ref applies to the return type and not
 to the this pointer (and that function doesn't even have a this pointer,
 since it's not a member function).


 It's an attribute on the return type.


 There is no such thing. Types cannot have attributes.


 Sure they can. e.g.

 auto func(ref int param) {...}

 The same with in and out. They apply to the type of the parameter without
 actually being part of it.


 So, it doesn't make sense for it to be on the right. That would be
 like having the const on the right-hand side of a member function apply to
 the return type rather than the function itself.
 ...


 You have it backwards.


 It looks to me like the compiler is treating ref in a schizophrenic manner
 given that when it's used on a parameter, it treats it as part of the
 parameter, whereas with the return type, it's treating it as a function
 attribute instead of associating it with the return type. I'd guess that
 that stems from the fact that while ref is really supposed to be associated
 with the type, it's not actually part of the type.


Bingo! But it's much deeper than that. ref is a disaster. Use it to
any real extent; in particular, binding (or fabricating bindings) to
extern(C++) code, and you will see just how fast it breaks down in the
most horrific of ways (almost always resulting in text mixin). I would
fork D just to fix ref!
Aug 04 2016
parent reply Seb <seb wilzba.ch> writes:
On Thursday, 4 August 2016 at 12:44:49 UTC, Manu wrote:

 On 29 July 2016 at 16:51, Jonathan M Davis via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 [...]


 Bingo! But it's much deeper than that. ref is a disaster. Use 
 it to
 any real extent; in particular, binding (or fabricating 
 bindings) to
 extern(C++) code, and you will see just how fast it breaks down 
 in the
 most horrific of ways (almost always resulting in text mixin). 
 I would
 fork D just to fix ref!


I might sound stupid, but there is a better way than forking:

1) Work out a _good_ D improvement proposal and submit it for 
review to https://github.com/dlang/DIPs (exact process is 
explained over there)
2) Once accepted, send a PR
Aug 04 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5 August 2016 at 03:33, Seb via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Thursday, 4 August 2016 at 12:44:49 UTC, Manu wrote:

 On 29 July 2016 at 16:51, Jonathan M Davis via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 [...]



 Bingo! But it's much deeper than that. ref is a disaster. Use it to
 any real extent; in particular, binding (or fabricating bindings) to
 extern(C++) code, and you will see just how fast it breaks down in the
 most horrific of ways (almost always resulting in text mixin). I would
 fork D just to fix ref!



 I might sound stupid, but there is a better way than forking:

 1) Work out a _good_ D improvement proposal and submit it for review to
 https://github.com/dlang/DIPs (exact process is explained over there)
 2) Once accepted, send a PR


Solution is trivial; ref is part of the type. Of course this will
break a ton of code, and it's strongly resisted. It'll never happen

_<
Aug 05 2016
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/5/16 6:17 AM, Manu via Digitalmars-d wrote:

 On 5 August 2016 at 03:33, Seb via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Thursday, 4 August 2016 at 12:44:49 UTC, Manu wrote:

 On 29 July 2016 at 16:51, Jonathan M Davis via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 [...]



 Bingo! But it's much deeper than that. ref is a disaster. Use it to
 any real extent; in particular, binding (or fabricating bindings) to
 extern(C++) code, and you will see just how fast it breaks down in the
 most horrific of ways (almost always resulting in text mixin). I would
 fork D just to fix ref!



 I might sound stupid, but there is a better way than forking:

 1) Work out a _good_ D improvement proposal and submit it for review to
 https://github.com/dlang/DIPs (exact process is explained over there)
 2) Once accepted, send a PR


 Solution is trivial; ref is part of the type. Of course this will
 break a ton of code, and it's strongly resisted. It'll never happen

 _<




Another solution is to make a Ref smart pointer type. But there isn't 
enough support from the language yet.

-Steve
Aug 05 2016
prev sibling next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 7/29/16 12:44 AM, Timon Gehr wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other function
 attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


Yes. Worst that happens is it doesn't get accepted :)

-Steve
Jul 29 2016
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Friday, 29 July 2016 at 04:44:16 UTC, Timon Gehr wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other 
 function attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


That doesn't fix the above mentioned problem, as ref bind to the 
variable rather than the type.

postfix ref is not accepted because it can only apply to the type 
I guess. This is one more example of design being unprincipled to 
boot.
Jul 29 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 30.07.2016 04:49, deadalnix wrote:

 On Friday, 29 July 2016 at 04:44:16 UTC, Timon Gehr wrote:

 My parser accepts the following:

 int function(int,int)ref functionPointer;

 I wasn't really aware that this was illegal in DMD. (Other function
 attributes, such as pure, are accepted.)

 In fact, even the following is disallowed:
 int foo(int)ref{}


 Should I file an enhancement request?


 That doesn't fix the above mentioned problem, as ref bind to the
 variable rather than the type.


How does it not fix the problem? int function(int,int)ref is the type of 
a ref-return function.


 postfix ref is not accepted because it can only apply to the type I
 guess.


How does it apply to the type?

This is one more example of design being unprincipled to boot.

Well, it's more or less reasonable save for the parser inconsistencies.
Jul 30 2016
parent deadalnix <deadalnix gmail.com> writes:
On Saturday, 30 July 2016 at 07:10:30 UTC, Timon Gehr wrote:

 How does it not fix the problem? int function(int,int)ref is 
 the type of a ref-return function.


Because storage class binding are already ambiguous and


 postfix ref is not accepted because it can only apply to the 
 type I
 guess.


 How does it apply to the type?

 This is one more example of design being unprincipled to boot.

 Well, it's more or less reasonable save for the parser 
 inconsistencies.
Jul 30 2016
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/28/2016 1:33 AM, Ethan Watson wrote:

 I also assume "lodge a bug" will be the response to these.


Indeed. That's the process.



 2) Expansion of code (static foreach, templates) is slow to the point where
 string mixins are a legitimate compile-time optimisation


https://issues.dlang.org/show_bug.cgi?id=16330
Jul 28 2016
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Thursday, 28 July 2016 at 08:33:22 UTC, Ethan Watson wrote:

 This also isn't the only use case I have. I'm a game engine 
 programmer. We write a lot of abstracted interfaces with 
 platform specific implementations. I know, I know, version(X){} 
 your code, right? But that's not how everyone works. Some 
 implementations really do require their own file for 
 maintenance and legal purposes.


The usual idea for PAL structure is to put implementations in 
separate folders:
src/lin/pal/utils.d - module pal.utils;
src/win/pal/utils.d - module pal.utils;
Then you can just import pal.utils; and invoke the compiler with 
-Isrc/lin option.
Jul 28 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-07-28 10:33, Ethan Watson wrote:


 4) Forward declaring a function prototype means I can never declare that
 function elsewhere (say, for example, with a mixin)


You mean like this:

import std.stdio;

void foo();

void foo()
{
     writeln("asd");
}

void main()
{
     foo();
}

That works for me [1]. It was reported by Manu and fixed in 2012 [2].

[1] https://dpaste.dzfl.pl/c6b9355158cf
[2] https://issues.dlang.org/show_bug.cgi?id=8108

-- 
/Jacob Carlborg
Aug 04 2016
parent reply Ethan Watson <gooberman gmail.com> writes:
On Thursday, 4 August 2016 at 11:41:00 UTC, Jacob Carlborg wrote:

 That works for me [1]. It was reported by Manu and fixed in 
 2012 [2].


I did some more experimenting, and it turns out that the problem 
is when the declaration and definition have different linkage. 
Being C++ functions means that all the functions are declared 
with extern( C++ ), but the mixin generates essentially extern( D 
) functions.

And a bit more prodding and poking, and I found some problems 
with UDAs - specifically, adding them in the declaration but not 
having them in the definition. We use UDAs extensively to mark up 
functions for binding. Depending on when my function collector 
template instantiates at compile time determines whether I can 
see the UDAs or not.

So that's technically a bug. But. Before I go running off to the 
Bugzilla Walter made. Should a user declaring and then later 
defining a function be forced to 100% replicate that function 
definition before defining it? If yes, then the compiler needs to 
error. If no, then there'll need to be some rules made up for it 
because I can already see how that will be open to abuse.
Aug 05 2016
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Friday, 5 August 2016 at 08:15:37 UTC, Ethan Watson wrote:

 So that's technically a bug. But. Before I go running off to 
 the Bugzilla Walter made. Should a user declaring and then 
 later defining a function be forced to 100% replicate that 
 function definition before defining it? If yes, then the 
 compiler needs to error. If no, then there'll need to be some 
 rules made up for it because I can already see how that will be 
 open to abuse.


i think that actual declaration should match in signature *only*; 
linkage and UDAs should be taken from prototype declaration. some 
rationale: this feature is useful mostly for binding generators 
(the only times i've seen it requested were for binding 
generators! ;-), so let's make programmer's lives easier here. 
checking for actual arg and return types is ok (as D doesn't have 
`()` syntax for "any args", we can't do "wildcard matching"), but 
there is absolutely no need to repeat other information.

let's hope that other people will agree, or the feature will stay 
barely useful, as it is now. ;-)
Aug 05 2016
parent ketmar <ketmar ketmar.no-ip.org> writes:
On Friday, 5 August 2016 at 08:40:04 UTC, ketmar wrote:
p.s. the only "gotcha" i see is if prototyp declaration is found 
*after* the actual function was declared. as D code should not 
depend of declaration order, this should be either error, or 
(better!) use prototype to "fix" previous definition (possibly 
with warning).
Aug 05 2016