www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Usability of "allMembers and derivedMembers traits now only return

reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

I don't agree with the current solution:

   http://dlang.org/changelog/2.071.2.html#traits-members-visibility

Modules should be able to use library templates without needing to mix 
them in first.

Do you think the solution in the change log usable? I don't think so 
because silently skipping my private members is an unexpected behavior 
that will cause bugs.

Further, do I understand the example right? Am I supposed to mixin the 
same template twice for two different types? The following code which 
adds another struct does not compile:

import std.stdio;
import std.traits;

enum UDA;
struct S
{
      UDA int visible;
      UDA private int invisible;
}

// only returns symbols visible from std.traits
static assert(getSymbolsByUDA!(S, UDA).length == 1);
// mixin the template instantiation, using a name to avoid namespace 
pollution
mixin getSymbolsByUDA!(S, UDA) symbols;
// as the template is instantiated in the current scope, it can see 
private members
static assert(symbols.getSymbolsByUDA.length == 2);

// --- The following is added by Ali: ---

struct S2 {
      UDA int s2;
}

mixin getSymbolsByUDA!(S2, UDA) symbolsS2;    // COMPILATION ERROR:
// Error: mixin deneme.getSymbolsByUDA!(S2, UDA) TList isn't a template

static assert(symbolsS2.getSymbolsByUDA.length == 1);

void main() {
     foreach (i; 0 .. symbols.getSymbolsByUDA.length) {
         // ...
     }
}

I can't wrap my head around the fact that a library template called by 
my module cannot see my private members.

Ali
Aug 30 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

   
 http://dlang.org/changelog/2.071.2.html#traits-members-visibility

 Modules should be able to use library templates without needing 
 to mix them in first.

 Do you think the solution in the change log usable? I don't 
 think so because silently skipping my private members is an 
 unexpected behavior that will cause bugs.

 Further, do I understand the example right? Am I supposed to 
 mixin the same template twice for two different types? The 
 following code which adds another struct does not compile:

 import std.stdio;
 import std.traits;

 enum UDA;
 struct S
 {
      UDA int visible;
      UDA private int invisible;
 }

 // only returns symbols visible from std.traits
 static assert(getSymbolsByUDA!(S, UDA).length == 1);
 // mixin the template instantiation, using a name to avoid 
 namespace pollution
 mixin getSymbolsByUDA!(S, UDA) symbols;
 // as the template is instantiated in the current scope, it can 
 see private members
 static assert(symbols.getSymbolsByUDA.length == 2);

 // --- The following is added by Ali: ---

 struct S2 {
      UDA int s2;
 }

 mixin getSymbolsByUDA!(S2, UDA) symbolsS2;    // COMPILATION 
 ERROR:
 // Error: mixin deneme.getSymbolsByUDA!(S2, UDA) TList isn't a 
 template

 static assert(symbolsS2.getSymbolsByUDA.length == 1);

 void main() {
     foreach (i; 0 .. symbols.getSymbolsByUDA.length) {
         // ...
     }
 }

 I can't wrap my head around the fact that a library template 
 called by my module cannot see my private members.

 Ali


Yes I agree, a change of the specifications in a dot release 
seems a bit extreme, especially since there was a deeper problem 
**before** the broken imports were fixed.

This problem was discovered a while back when the library traits 
related to UDAs were added to phobos.

The little story began here: 
https://issues.dlang.org/show_bug.cgi?id=15335. Then the proposal 
to give super powers to certain traits verbs: 
https://issues.dlang.org/show_bug.cgi?id=15371.
Aug 30 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Tuesday, 30 August 2016 at 22:46:58 UTC, Basile B. wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 v2.071.2-b3 is bringing a change for this bug:


 Yes I agree, a change of the specifications in a dot release 
 seems a bit extreme, especially since there was a deeper 
 problem **before** the broken imports were fixed.

 This problem was discovered a while back when the library 
 traits related to UDAs were added to phobos.

 The little story began here: 
 https://issues.dlang.org/show_bug.cgi?id=15335. Then the 
 proposal to give super powers to certain traits verbs: 
 https://issues.dlang.org/show_bug.cgi?id=15371.


To be clear, the logic I see for traits "getMember", "allMember", 
"getOverloads", "derivedMembers" (etc, all the traits that might 
be today limited by the protection attribute) is:

allow them to see everything, then use "getProtection" if you 
wanna be conform with the protection attributes.

Maybe it's worth a DIP ? Casual and informal discussions have 
**failed**.

I see now the answer that mentions ".tupleof". I see no valid 
logic that would allow ".tupleof" to see everything and not the 
traits.
Aug 31 2016
parent reply Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 31 August 2016 at 08:06:05 UTC, Basile B. wrote:

 allow them to see everything, then use "getProtection" if you 
 wanna be conform with the protection attributes.


That's how it used to work, but getProtection would fail if the 
symbol wasn't public. Which led to me using a workaround to 
something of this effect:

enum PrivacyLevel : string
{
	Public			= "public",
	Private			= "private",
	Protected		= "protected",
	Export			= "export",
	Package			= "package",
	Inaccessible		= "inaccessible"
};
//----------------------------------------------------------------------------

template PrivacyOf( alias symbol )
{
	static if( __traits( compiles, __traits( getProtection, symbol ) 
) )
	{
		enum PrivacyOf = cast(PrivacyLevel) __traits( getProtection, 
symbol );
	}
	else
	{
		enum PrivacyOf = PrivacyLevel.Inaccessible;
	}
}
//----------------------------------------------------------------------------

Still not an ideal solution - because if I'm trying to serialise 
and deserialise everything in between module reloads I still need 
to do the .tupleof method to get all data members; and if I want 
to define privacy levels for functions I'm automatically binding 
from C++ I need to muddy those waters with UDAs etc.
Aug 31 2016
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 31 August 2016 at 08:33:28 UTC, Ethan Watson wrote:

 That's how it used to work, but getProtection would fail if the 
 symbol wasn't public. Which led to me using a workaround to 
 something of this effect:


Yeah, I kinda regret the design of getProtection (which is 
basically 100% my fault, I implemented it myself and pushed it 
through without thinking about private - I was only interested in 
public vs export for my personal use case...), but if getMember 
worked on private things too, getProtection would never have to 
return inaccessible and it would be pretty again.
Aug 31 2016
prev sibling next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8/31/16, Ali Çehreli via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 mixin getSymbolsByUDA!(S, UDA) symbols;


This is such a bizarre workaround to be listed in the changelog since
mixing in non-mixin templates is not an official feature (am I
wrong?). getSymbolsByUDA is a template, not a mixin template.
Aug 30 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-08-31 01:01, Andrej Mitrovic via Digitalmars-d wrote:


 This is such a bizarre workaround to be listed in the changelog since
 mixing in non-mixin templates is not an official feature (am I
 wrong?).


Yes. Originally one could not use the "mixin" keyword in front of a 
template. There was no difference between mixin templates and non-mixin 
templates, on the declaration site. Later the language was changed to 
allow to put "mixin" in front of a template. That restricted all mixin 
templates to only be used as mixin templates. It's still possible to 
mixin non-mixin templates, most likely to avoid breaking existing code.

-- 
/Jacob Carlborg
Aug 30 2016
prev sibling next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8/31/16, Ali Çehreli via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 v2.071.2-b3 is bringing a change for this bug:

    https://issues.dlang.org/show_bug.cgi?id=15907


PSA: If all one cares about are UDAs for fields and not functions then
.tupleof is still a viable workaround. Working example:

-----
import getSymbols;

enum UDA;

struct S
{
     UDA float visible;

    float dont_care;

    private  UDA int invisible;
}

void main()
{
    static assert(getSymbolsByUDA!(S, UDA).length == 2);
    static assert(getSymbolsByUDA!(S, UDA).stringof == "tuple(visible,
invisible)");
}
-----

-----
module getSymbols;

import std.meta : AliasSeq;
import std.traits : staticIndexOf;

template getSymbolsByUDA ( T, alias uda )
{
    alias getSymbolsByUDA = getSymbolsByUDAImpl!(T, uda);
}

template getSymbolsByUDAImpl ( T, alias uda, size_t idx = 0 )
{
    static if (idx + 1 < T.tupleof.length)
    {
        static if (hasUDA!(T.tupleof[idx], uda))
            alias Field = AliasSeq!(T.tupleof[idx]);
        else
            alias Field = AliasSeq!();

        alias getSymbolsByUDAImpl = AliasSeq!(Field,
            getSymbolsByUDAImpl!(T, uda, idx + 1));
    }
    else
    {
        static if (hasUDA!(T.tupleof[idx], uda))
            alias getSymbolsByUDAImpl = AliasSeq!(T.tupleof[idx]);
        else
            alias getSymbolsByUDAImpl = AliasSeq!();
    }
}

template hasUDA ( alias field, alias uda )
{
    enum hasUDA = staticIndexOf!(uda, __traits(getAttributes, field)) != -1;
}
-----

This is the reason msgpack-d still works and wasn't broken by the change.
Aug 30 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-08-31 02:16, Andrej Mitrovic via Digitalmars-d wrote:


 PSA: If all one cares about are UDAs for fields and not functions then
 .tupleof is still a viable workaround. Working example:

 -----
 import getSymbols;

 enum UDA;

 struct S
 {
      UDA float visible;

     float dont_care;

     private  UDA int invisible;
 }

 void main()
 {
     static assert(getSymbolsByUDA!(S, UDA).length == 2);
     static assert(getSymbolsByUDA!(S, UDA).stringof == "tuple(visible,
 invisible)");
 }
 -----

 -----
 module getSymbols;

 import std.meta : AliasSeq;
 import std.traits : staticIndexOf;

 template getSymbolsByUDA ( T, alias uda )
 {
     alias getSymbolsByUDA = getSymbolsByUDAImpl!(T, uda);
 }

 template getSymbolsByUDAImpl ( T, alias uda, size_t idx = 0 )
 {
     static if (idx + 1 < T.tupleof.length)
     {
         static if (hasUDA!(T.tupleof[idx], uda))
             alias Field = AliasSeq!(T.tupleof[idx]);
         else
             alias Field = AliasSeq!();

         alias getSymbolsByUDAImpl = AliasSeq!(Field,
             getSymbolsByUDAImpl!(T, uda, idx + 1));
     }
     else
     {
         static if (hasUDA!(T.tupleof[idx], uda))
             alias getSymbolsByUDAImpl = AliasSeq!(T.tupleof[idx]);
         else
             alias getSymbolsByUDAImpl = AliasSeq!();
     }
 }

 template hasUDA ( alias field, alias uda )
 {
     enum hasUDA = staticIndexOf!(uda, __traits(getAttributes, field)) != -1;
 }
 -----

 This is the reason msgpack-d still works and wasn't broken by the change.


Ah, nice workaround. Last time I tried to use __traits(getAttributes) 
with a "tupleof expression" it didn't work.

-- 
/Jacob Carlborg
Aug 30 2016
prev sibling next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
yeah, the whole feature smells for me. the sole need of mixin 
hack indicates that something is very wrong here. i never ever 
needed that for normal D code. and now suddenly i have to 
remember that some thing is a template, that it needs mixing it 
to work properly, etc.

all this mess should be resolved in compiler by assigning 
template *two* visibility scopes: one is template's "normal" 
scope (so it can see symbols from it's originating module), and 
second is it's "instantiation" scope. after all, this is exactly 
what programmer is exepecting. current solution is not a solution 
at all, it's a hacky workaround promoted to "official technique".
Aug 30 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 31 August 2016 at 05:33:50 UTC, ketmar wrote:

 all this mess should be resolved in compiler by assigning 
 template *two* visibility scopes: one is template's "normal" 
 scope (so it can see symbols from it's originating module), and 
 second is it's "instantiation" scope. after all, this is 
 exactly what programmer is exepecting. current solution is not 
 a solution at all, it's a hacky workaround promoted to 
 "official technique".


This would require a new instance for every template 
instantiation. The instantiation scope does not leak into the 
template by design, it allows us to cache instantiations and 
greatly speed up compilation.
Sep 03 2016
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Saturday, 3 September 2016 at 14:48:54 UTC, Martin Nowak wrote:

 On Wednesday, 31 August 2016 at 05:33:50 UTC, ketmar wrote:

 all this mess should be resolved in compiler by assigning 
 template *two* visibility scopes: one is template's "normal" 
 scope (so it can see symbols from it's originating module), 
 and second is it's "instantiation" scope. after all, this is 
 exactly what programmer is exepecting. current solution is not 
 a solution at all, it's a hacky workaround promoted to 
 "official technique".


 This would require a new instance for every template 
 instantiation. The instantiation scope does not leak into the 
 template by design, it allows us to cache instantiations and 
 greatly speed up compilation.


just a wrapper class, which will hold the actual instantiation 
and a scope. most of the code should "pass thru" the wrapper (i 
thing that `alias this` can be used for that, but have to check 
it), yet `__traits` can use additional info. sure, that will 
require some work, but it's not that impossible, and should not 
ruin caching. yeah, we will waste additional 16/32 bytes per 
template instance this way. not that much, and it will solve the 
problem in most natural way.
Sep 03 2016
parent Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 15:48:24 UTC, ketmar wrote:

 just a wrapper class, which will hold the actual instantiation 
 and a scope. most of the code should "pass thru" the wrapper (i 
 thing that `alias this` can be used for that, but have to check 
 it), yet `__traits` can use additional info. sure, that will 
 require some work, but it's not that impossible, and should not 
 ruin caching. yeah, we will waste additional 16/32 bytes per 
 template instance this way. not that much, and it will solve 
 the problem in most natural way.


Well, if you're making a difference based on the instantiation 
scope inside the template, then you're are leaking it into the 
template, do require a new instance for every instantiation, are 
ruining caching, do require unique mangling and redundant codegen.
We have mixin templates exactly for the purpose of instantiating 
them in the origin scope.
Sep 03 2016
prev sibling next sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I don't agree with the current solution:


I'm somewhat surprised myself that "allMembers doesn't return all 
members" needs highlighting.

Why not have a new trait "allVisibleMembers" and just fix the 
privacy issues?
Aug 31 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 31 August 2016 at 08:36:37 UTC, Ethan Watson wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I don't agree with the current solution:


 I'm somewhat surprised myself that "allMembers doesn't return 
 all members" needs highlighting.

 Why not have a new trait "allVisibleMembers" and just fix the 
 privacy issues?


nice idea, but this doesn't change the fact that the traits that 
access the results of the "omniscient" allMember must be tweaked 
to access everything.
Aug 31 2016
parent reply Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 31 August 2016 at 09:25:52 UTC, Basile B. wrote:

 nice idea, but this doesn't change the fact that the traits 
 that access the results of the "omniscient" allMember must be 
 tweaked to access everything.


I'm okay with this. My PrivacyLevel workaround does exactly this 
in fact.

But I would like to be able to read (and write) all members of a 
class without needing to mixin a template and without having to 
resort to .tupleof. Use case here is the extensive struct use we 
have, if we want them to match C++ exactly then that's where 
mixins become potentially hairy.
Aug 31 2016
parent Ethan Watson <gooberman gmail.com> writes:
On Wednesday, 31 August 2016 at 09:30:43 UTC, Ethan Watson wrote:

 I'm okay with this. My PrivacyLevel workaround does exactly 
 this in fact.


I keep forgetting that I'm all open sourced now and can just link 
directly to the full example.

https://github.com/Remedy-Entertainment/binderoo/blob/master/binderoo_client/d/src/binderoo/objectprivacy.d
Aug 31 2016
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Ugh, it really should just give everything and have getMember 
bypass it. That won't even break any code!
Aug 31 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:

 Ugh, it really should just give everything and have getMember 
 bypass it. That won't even break any code!


you're right. "allMembers" means "all" after all. Another reason 
why the idea of "allVisibleMembers" is good. Puristes will be 
able to use this traits without messing with "getProtection".
Aug 31 2016
parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 31 August 2016 at 13:29:52 UTC, Basile B. wrote:

 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe 
 wrote:

 Ugh, it really should just give everything and have getMember 
 bypass it. That won't even break any code!


 you're right. "allMembers" means "all" after all. Another 
 reason why the idea of "allVisibleMembers" is good. Puristes 
 will be able to use this traits without messing with 
 "getProtection".


https://github.com/dlang/DIPs/pull/39

Co-authors welcome.
Sep 01 2016
parent Martin Nowak <code dawg.eu> writes:
On Thursday, 1 September 2016 at 19:30:41 UTC, Basile B. wrote:

 https://github.com/dlang/DIPs/pull/39

 Co-authors welcome.


Slow down a bit until we've finally decided out how to resolve 
the problems.
Sep 03 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:

 Ugh, it really should just give everything and have getMember 
 bypass it. That won't even break any code!


It will, e.g. having getMember bypass protection means vibe.d 
would now serialize private members 
https://github.com/rejectedsoftware/vibe.d/blob/c1180791de61d0f8c9bfb584c2551a5b64627e32/source/vibe/internal/meta/traits.d#L146.
Until now getMember was just a DotIdExp (i.e. 
`T.indent`/`var.ident`), so it was used to test whether sth. can 
be accessed.
Sep 03 2016
next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/3/16, Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:

 Ugh, it really should just give everything and have getMember
 bypass it. That won't even break any code!


 It will, e.g. having getMember bypass protection means vibe.d
 would now serialize private members


Then just add a check in vibe.d itself to avoid serializing private
members. You can still call getProtection on the symbol and skip
serializing it.

Alternatively you can use UDAs so you can mark which fields should or
shouldn't be serializible. For example
https://github.com/msgpack/msgpack-d/blob/6046808c2e678e27cb2e2d9314241c361a6fd0ae/src/msgpack/attribute.d#L21

The bottom line is with restricting access to private symbols you have
no choice on the matter, while allowing access lets you specialize
whether to ignore the symbols or not.
Sep 03 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 6:39 PM, Andrej Mitrovic via Digitalmars-d wrote:

 On 9/3/16, Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:

 Ugh, it really should just give everything and have getMember
 bypass it. That won't even break any code!


 It will, e.g. having getMember bypass protection means vibe.d
 would now serialize private members


 Then just add a check in vibe.d itself to avoid serializing private
 members. You can still call getProtection on the symbol and skip
 serializing it.

 Alternatively you can use UDAs so you can mark which fields should or
 shouldn't be serializible. For example
 https://github.com/msgpack/msgpack-d/blob/6046808c2e678e27cb2e2d9314241c361a6fd0ae/src/msgpack/attribute.d#L21

 The bottom line is with restricting access to private symbols you have
 no choice on the matter, while allowing access lets you specialize
 whether to ignore the symbols or not.


I didn't follow this closely, but clearly we need some means to access 
private members for introspection purposes. I hope we don't paint 
ourselves out of that corner. -- Andrei
Sep 03 2016
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Saturday, September 03, 2016 19:30:54 Andrei Alexandrescu via Digitalmars-d 
wrote:

 On 9/3/16 6:39 PM, Andrej Mitrovic via Digitalmars-d wrote:

 On 9/3/16, Martin Nowak via Digitalmars-d <digitalmars-d puremagic.com> 




wrote:

 On Wednesday, 31 August 2016 at 13:12:30 UTC, Adam D. Ruppe wrote:

 Ugh, it really should just give everything and have getMember
 bypass it. That won't even break any code!


 It will, e.g. having getMember bypass protection means vibe.d
 would now serialize private members


 Then just add a check in vibe.d itself to avoid serializing private
 members. You can still call getProtection on the symbol and skip
 serializing it.

 Alternatively you can use UDAs so you can mark which fields should or
 shouldn't be serializible. For example
 https://github.com/msgpack/msgpack-d/blob/6046808c2e678e27cb2e2d9314241c36
 1a6fd0ae/src/msgpack/attribute.d#L21

 The bottom line is with restricting access to private symbols you have
 no choice on the matter, while allowing access lets you specialize
 whether to ignore the symbols or not.


 I didn't follow this closely, but clearly we need some means to access
 private members for introspection purposes. I hope we don't paint
 ourselves out of that corner. -- Andrei


Well, that certainly seems to be exactly what we're doing with the changes
that Ali is complaining about.

Previously __traits(allMembers, T) listed _all_ members, whereas now it wil
only list the ones that are accessible. The same for
__traits(derivedMembers, T). So, they'll still give you the private members
if you use them inside the module in question but not elsewhere.

I would have thought that the way to handle this would be to have them list
all members regardless of access level and then to have code check for the
access level if it cares. I would have thought that doing stuff like looking
at private symbols would be perfectly okay with no errors or deprecation
warnings and that it would just be _accessing_ the symbol that would be
disallowed and would require you to check for in code that used
introspection. But clearly, there must be something that I misunderstand
here, because that's not at all where this seems to be going.

I do get the feeling that too much of what we do with features like this is
ad-hoc without really designing things up front, and then we keep having to
tweak stuff as we go along in ways that aren't always particularly nice. And
this particular change seems to just be a reaction to problems that we ran
into with changing how imports work. And maybe it's the right choice, but
it sure doesn't seem like it.

- Jonathan M Davis
Sep 03 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 7:57 PM, Jonathan M Davis via Digitalmars-d wrote:

 Previously __traits(allMembers, T) listed _all_ members, whereas now it wil
 only list the ones that are accessible. The same for
 __traits(derivedMembers, T). So, they'll still give you the private members
 if you use them inside the module in question but not elsewhere.


That... doesn't sound good. I wonder how such important changes slip by 
Walter and myself unnoticed.

My thinking is that the plebes should be able to access things via the 
object.member syntax by obeying the usual visibility rules. But 
__traits(allMembers, T) should be the reflection backdoor that gives the 
savvy users total access, at the obvious cost of an awkward syntax.

The fact that __traits(allMembers, T) compiles in all cases and has 
DIFFERENT semantics depending on whether T is in the same vs. a 
different module is the deadly sign of poor language design.

Martin, any chance we can undo this change to the language?


Andrei
Sep 03 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei 
Alexandrescu wrote:

 Martin, any chance we can undo this change to the language?


The problem is deeper. If he undoes the change then what will 
happen again is that the results of allMembers won't be usable by 
the other traits, e.g getMember, because them are still limited 
by the protection attributes.
Sep 03 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 10:50 PM, Basile B. wrote:

 On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei Alexandrescu wrote:

 Martin, any chance we can undo this change to the language?


 The problem is deeper. If he undoes the change then what will happen
 again is that the results of allMembers won't be usable by the other
 traits, e.g getMember, because them are still limited by the protection
 attributes.


Where are the past discussions on this matter? -- Andrei
Sep 03 2016
parent Basile B. <b2.temp gmx.com> writes:
On Saturday, 3 September 2016 at 20:54:19 UTC, Andrei 
Alexandrescu wrote:

 On 9/3/16 10:50 PM, Basile B. wrote:

 On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei 
 Alexandrescu wrote:

 Martin, any chance we can undo this change to the language?


 The problem is deeper. If he undoes the change then what will 
 happen
 again is that the results of allMembers won't be usable by the 
 other
 traits, e.g getMember, because them are still limited by the 
 protection
 attributes.


 Where are the past discussions on this matter? -- Andrei


The change in 2.071.2-beta3 to allMembers is designed to fix

https://issues.dlang.org/show_bug.cgi?id=15907

which happens because of a __traits(getMember,...) on the results 
of a __traits(allMembers,...).
Sep 03 2016
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-03 22:40, Andrei Alexandrescu wrote:


 That... doesn't sound good. I wonder how such important changes slip by
 Walter and myself unnoticed.


Here's the PR that introduced the change: 
https://github.com/dlang/dmd/pull/6078

-- 
/Jacob Carlborg
Sep 03 2016
next sibling parent reply Ethan Watson <gooberman gmail.com> writes:
On Saturday, 3 September 2016 at 21:54:24 UTC, Jacob Carlborg 
wrote:

 Here's the PR that introduced the change: 
 https://github.com/dlang/dmd/pull/6078


I'm certainly not going to upgrade to the next DMD if this change 
is retained. allMembers not returning all members makes 
introspection entirely useless when it comes to Binderoo.

The wrong conclusions were made from that bug to begin with it 
seems. allMembers should return all members. getProtection should 
report on the protection of a symbol *regardless* of whether 
getMember will succeed or not (this is currently why I have my 
PrivacyOf workaround - to stop the compiler crashing when using a 
template instead of a mixin template to do introspection).

getMember itself, well, I'd honestly prefer if there was a way to 
get to members without having to correlate with .tupleof as it 
will simplify Binderoo code. The .tupleof method doesn't help me 
when it comes to introspecting private/protected functions for 
C++ binding.
Sep 03 2016
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/03/2016 03:13 PM, Ethan Watson wrote:

 On Saturday, 3 September 2016 at 21:54:24 UTC, Jacob Carlborg wrote:



 allMembers not returning all members makes introspection
 entirely useless when it comes to Binderoo.


Same problem with Weka's code base...


 getMember itself, well, I'd honestly prefer if there was a way to get to
 members without having to correlate with .tupleof as it will simplify
 Binderoo code.


Again, same problem at Weka...

Protection attributes are for protecting programmers from implementation 
changes. If I reach for allMembers or getMember, it's an explicit way of 
saying "I don't care about the consequences."

Ali
Sep 07 2016
next sibling parent Chris Wright <dhasenan gmail.com> writes:
On Wed, 07 Sep 2016 12:21:40 -0700, Ali Çehreli wrote:

 Protection attributes are for protecting programmers from implementation
 changes. If I reach for allMembers or getMember, it's an explicit way of
 saying "I don't care about the consequences."


If it's just about implementation changes, allMembers / getMember is a 
way of inspecting types to get what currently exists. It will continue to 
be correct in the face of changing fields.
Sep 07 2016
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/7/16 9:21 PM, Ali Çehreli wrote:

 On 09/03/2016 03:13 PM, Ethan Watson wrote:

 On Saturday, 3 September 2016 at 21:54:24 UTC, Jacob Carlborg wrote:



 allMembers not returning all members makes introspection
 entirely useless when it comes to Binderoo.


 Same problem with Weka's code base...


 getMember itself, well, I'd honestly prefer if there was a way to get to
 members without having to correlate with .tupleof as it will simplify
 Binderoo code.


 Again, same problem at Weka...

 Protection attributes are for protecting programmers from implementation
 changes. If I reach for allMembers or getMember, it's an explicit way of
 saying "I don't care about the consequences."


Martin has a PR to improve on this, could you all please take a look? I 
can't really do much until Sunday, I'm on a WiFi connection slower than 
a herd of turtles, and which only works if I hold my laptop over my 
head. So I'll click "Send" on this then push the roof with my laptop. -- 
Andrei
Sep 08 2016
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/3/16 11:54 PM, Jacob Carlborg wrote:

 On 2016-09-03 22:40, Andrei Alexandrescu wrote:


 That... doesn't sound good. I wonder how such important changes slip by
 Walter and myself unnoticed.


 Here's the PR that introduced the change:
 https://github.com/dlang/dmd/pull/6078


Thanks. -- Andrei
Sep 03 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/3/2016 2:54 PM, Jacob Carlborg wrote:

 On 2016-09-03 22:40, Andrei Alexandrescu wrote:


 That... doesn't sound good. I wonder how such important changes slip by
 Walter and myself unnoticed.


 Here's the PR that introduced the change:
https://github.com/dlang/dmd/pull/6078


Looks like I'm to blame as I pulled it.
Sep 04 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 20:40:57 UTC, Andrei 
Alexandrescu wrote:

 On 9/3/16 7:57 PM, Jonathan M Davis via Digitalmars-d wrote:

 Previously __traits(allMembers, T) listed _all_ members, 
 whereas now it wil
 only list the ones that are accessible. The same for
 __traits(derivedMembers, T). So, they'll still give you the 
 private members
 if you use them inside the module in question but not 
 elsewhere.


 That... doesn't sound good. I wonder how such important changes 
 slip by Walter and myself unnoticed.


It didn't slip, but I wish Walter had at least stated his opinion 
on the PR before merging.


 My thinking is that the plebes should be able to access things 
 via the object.member syntax by obeying the usual visibility 
 rules. But __traits(allMembers, T) should be the reflection 
 backdoor that gives the savvy users total access, at the 
 obvious cost of an awkward syntax.


As explained several times here and in the announce thread, 
private members have never been accessible, other than 
introspecting attributes, and making them accessible comes with a 
performance cost and a fairly big language change.
So the real question is, why do we need introspection without 
access, and can we handle that few cases with mixin templates.
If we really need introspection of private members than we might 
need to go back to the drawing board and modify the visibility 
concept introduced with DIP22.
Sep 03 2016
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 4 September 2016 at 03:14:18 UTC, Martin Nowak wrote:

 It didn't slip, but I wish Walter had at least stated his 
 opinion on the PR before merging.


 My thinking is that the plebes should be able to access things 
 via the object.member syntax by obeying the usual visibility 
 rules. But __traits(allMembers, T) should be the reflection 
 backdoor that gives the savvy users total access, at the 
 obvious cost of an awkward syntax.


 As explained several times here and in the announce thread, 
 private members have never been accessible, other than 
 introspecting attributes, and making them accessible comes with 
 a performance cost and a fairly big language change.
 So the real question is, why do we need introspection without 
 access, and can we handle that few cases with mixin templates.
 If we really need introspection of private members than we 
 might need to go back to the drawing board and modify the 
 visibility concept introduced with DIP22.


While I do understand, that there could be a potential 
performance when  private members could be changed around because 
they are not visible form outside.

I fail to see how we would take advantage of that without 
breaking our object-model.
Sep 03 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 4 September 2016 at 03:21:05 UTC, Stefan Koch wrote:

 While I do understand, that there could be a potential 
 performance when  private members could be changed around 
 because they are not visible form outside.

 I fail to see how we would take advantage of that without 
 breaking our object-model.


It's mostly about linkage of private methods.
Sep 03 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 04 Sep 2016 03:29:59 +0000
schrieb Martin Nowak <code dawg.eu>:


 On Sunday, 4 September 2016 at 03:21:05 UTC, Stefan Koch wrote:

 While I do understand, that there could be a potential 
 performance when  private members could be changed around 
 because they are not visible form outside.

 I fail to see how we would take advantage of that without 
 breaking our object-model.  


 
 It's mostly about linkage of private methods.


Allowing access to private fields / functions also means that
those members are part of the public API/ABI of a library. So changing
private members can break dependent libraries...
Sep 04 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-09-04 09:39, Johannes Pfau wrote:


 Allowing access to private fields / functions also means that
 those members are part of the public API/ABI of a library. So changing
 private members can break dependent libraries...


They've always been accessible using .tupleof and taking the address of 
a member, before the recent lookup rule changes.

-- 
/Jacob Carlborg
Sep 04 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 5:14 AM, Martin Nowak wrote:

 If we really need introspection of private members than we might need to
 go back to the drawing board and modify the visibility concept
 introduced with DIP22.


Thanks for answering. Yes, we really need introspection of private 
members. One way or another we need to support that. -- Andrei
Sep 04 2016
parent reply David Nadlinger <code klickverbot.at> writes:
On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu 
wrote:

 Thanks for answering. Yes, we really need introspection of 
 private members. One way or another we need to support that.


Do we, though? It's easy to state a general claim like this, but 
I find it hard to actually justify.

  — David
Sep 04 2016
next sibling parent David Nadlinger <code klickverbot.at> writes:
On Sunday, 4 September 2016 at 12:37:47 UTC, David Nadlinger 
wrote:

 Do we, though? It's easy to state a general claim like this, 
 but I find it hard to actually justify.


(And to pre-empt the collective freak-out, note that this 
statement is coming from somebody who has done quite a bit of 
pioneering serialisation/meta-programming work. — David)
Sep 04 2016
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 2:37 PM, David Nadlinger wrote:

 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu wrote:

 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.


 Do we, though? It's easy to state a general claim like this, but I find
 it hard to actually justify.


Static introspection is by far the most powerful feature of D. The last 
thing to do with it is hamstring it with nonsense like private :o).

Quoting elsethread:


 There are a few cases I can think of:

 * Serialization and deserialization, shallow and deep. These would need access
to the object layout, and possibly field names too (for cross-checking and
versioning purposes).

 * Binary saving, loading, and fixup (a subset of
serialization/deserialization) - that thing when you copy raw memory to a file
and then load it raw and fix pointers up.

 * Memory management and garbage collection: access to field types allows
efficient generic tracing.

 * Database interfacing, automated binding, object-relational mapping, etc. It
stands to reason that field names would be needed and fields would be private,
yet the database loader does have access to them.

 Of course the more important applications are those I can't yet imagine. The
way I see it introspection must have full power and unfettered access. So
definitely it must be able to pass through regular protection.



Andrei
Sep 04 2016
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 4 Sep 2016 14:54:33 +0200
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/4/16 2:37 PM, David Nadlinger wrote:

 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu
 wrote:  

 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.  


 Do we, though? It's easy to state a general claim like this, but I
 find it hard to actually justify.  


 
 Static introspection is by far the most powerful feature of D. The
 last thing to do with it is hamstring it with nonsense like
 private :o).
 
 Quoting elsethread:
 

 There are a few cases I can think of:

 * Serialization and deserialization, shallow and deep. These would
 need access to the object layout, and possibly field names too (for
 cross-checking and versioning purposes).

 * Binary saving, loading, and fixup (a subset of
 serialization/deserialization) - that thing when you copy raw
 memory to a file and then load it raw and fix pointers up.

 * Memory management and garbage collection: access to field types
 allows efficient generic tracing.

 * Database interfacing, automated binding, object-relational
 mapping, etc. It stands to reason that field names would be needed
 and fields would be private, yet the database loader does have
 access to them.

 Of course the more important applications are those I can't yet
 imagine. The way I see it introspection must have full power and
 unfettered access. So definitely it must be able to pass through
 regular protection.  


 
 
 Andrei


All these examples need access to private data fields only.
Most performance optimizations are only possible for private functions
anyway. So is there a use case to access/call private functions?
Sep 05 2016
parent Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Monday, 5 September 2016 at 07:48:11 UTC, Johannes Pfau wrote:

 Am Sun, 4 Sep 2016 14:54:33 +0200
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 9/4/16 2:37 PM, David Nadlinger wrote:

 [...]


 
 Static introspection is by far the most powerful feature of D. 
 The last thing to do with it is hamstring it with nonsense 
 like private :o).
 
 Quoting elsethread:
 

 [...]


 
 
 Andrei


 All these examples need access to private data fields only.
 Most performance optimizations are only possible for private 
 functions
 anyway. So is there a use case to access/call private functions?


One use case would be accessing UDA on private fields, to inspect 
which fields need to be injected with a dependency, in aggregate 
type, just like spring from java does with autowire annotation. 
You mark a private field with autowired, and spring calls 
associated setter for the field.

Does tupleof allow access to associated UDAs of contained fields, 
as well field names?
Sep 05 2016
prev sibling next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/04/2016 05:37 AM, David Nadlinger wrote:

 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu wrote:

 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.


 Do we, though? It's easy to state a general claim like this, but I find
 it hard to actually justify.

  — David


Let me try to reword what I've already said elsewhere in this thread.

As the user of a library I shouldn't need to know whether a template of 
that library uses __traits(allMembers) or not. Unfortunately, if 
__traits(allMembers) depends on access rights, I am forced to mixin 
*every* template in my code because I don't and should not know the 
template's implementation. If I don't know their implementation, I have 
to prepare myself for the worst and mixin *every* template. That's insanity.

Even if I know that they don't use __traits(allMembers) today, they may 
change their implementation in the future. So, I really have to mixin 
every template today.

Further, I have to be on my toes and watch every feature of every 
library in case they changed e.g. a function to a function template in a 
new release. Oh yes, I have to mixin that new template as well! (Again, 
because I can't be sure whether they use __traits(allMembers) or not.)

This new behavior and its mixin workaround is so insane to me that I 
can't even find ways to spell it out clearly. This behavior kills IFTI 
altogether because I don't know who uses __traits(allMembers). I can't 
rely on IFTI. I have to mixin every template instantiation myself. Crazy.

Ali
Sep 06 2016
prev sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/04/2016 05:37 AM, David Nadlinger wrote:

 On Sunday, 4 September 2016 at 12:33:07 UTC, Andrei Alexandrescu wrote:

 Thanks for answering. Yes, we really need introspection of private
 members. One way or another we need to support that.


 Do we, though? It's easy to state a general claim like this, but I find
 it hard to actually justify.

  — David


Let me flip the question: What harm can there be when I pass my type to 
a template and that template accesses the private members?

Ali
Sep 06 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 17:57:08 UTC, Jonathan M Davis 
wrote:

 I do get the feeling that too much of what we do with features 
 like this is ad-hoc without really designing things up front, 
 and then we keep having to tweak stuff as we go along in ways 
 that aren't always particularly nice.


That's not true, there was DIP22, and it explicitly mentions how 
traits should work.
The problem that was new and very hard to see, is that allowing 
traits to bypass visibility checks means we either can't replace 
access checks, must allow access of private members (w/ the cost 
tradeoffs mentioned in the announce thread), or come up with a 
different solution.
While it's fairly straightforward to skip visibility checks 
during traits lookups, I couldn't come up with a useful use-case 
while writing tests for that implementation, and after some more 
thought came up with the allMembers change.
Sep 03 2016
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-09-03 17:50, Martin Nowak wrote:


 It will, e.g. having getMember bypass protection means vibe.d would now
 serialize private members
 https://github.com/rejectedsoftware/vibe.d/blob/c1180791de61d0f8c9bfb584c2551a5b64627e32/source/vibe/internal/meta/traits.d#L146.

 Until now getMember was just a DotIdExp (i.e. `T.indent`/`var.ident`),
 so it was used to test whether sth. can be accessed.


Without looking at the how the rest of the code works, it already checks 
if a member is public or not [1]. Or are you saying that 
"isPublicMember" would return a different value?

[1] 
https://github.com/rejectedsoftware/vibe.d/blob/c1180791de61d0f8c9bfb584c2551a5b64627e32/source/vibe/internal/meta/traits.d#L159

-- 
/Jacob Carlborg
Sep 03 2016
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, August 30, 2016 15:24:12 Ali Çehreli via Digitalmars-d wrote:

 v2.071.2-b3 is bringing a change for this bug:

    https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

    http://dlang.org/changelog/2.071.2.html#traits-members-visibility

 Modules should be able to use library templates without needing to mix
 them in first.


Agreed. Having to mix stuff in for introspection is downright ugly. It makes
far more sense to provide everything and then check the attributes as
discussed elsewhere in this thread.

IMHO, having allMembers give different results depending on access level is
just plain broken.

I confess that I see no problem being able to examine symbols that are not
accessible due to the access level. They just shouldn't be usable or be
involved in overload sets. Looking at their declarations and attributes and
whatnot should be fine.

- Jonathan M Davis
Aug 31 2016
prev sibling next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 08/30/2016 03:24 PM, Ali Çehreli wrote:

 v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

   http://dlang.org/changelog/2.071.2.html#traits-members-visibility

 Modules should be able to use library templates without needing to mix
 them in first.

 Do you think the solution in the change log usable? I don't think so
 because silently skipping my private members is an unexpected behavior
 that will cause bugs.


Here is a regression caused by the above change:

interface I {
     void foo();

package:

     void foo(int);
}

string how(alias Base, alias func)() {
     return "";
}

import std.typecons;
class C : AutoImplement!(I, how) {
}

void main() {
     auto c = new C();
     c.foo();
     c.foo(42);    // COMPILATION ERROR:
// deneme.o: In function `_Dmain':
// deneme.d:(.text._Dmain+0x39): undefined reference to 
`_D6deneme1I3fooMFiZv'
// collect2: error: ld returned 1 exit status
}

WAT?

It's not reasonable for the user to somehow suspect that AutoImplement 
may be using a D feature (__traits(allMembers) in this case) which 
happens to not see foo(int). (Note that AutoImplement may see foo(int) 
today but not tomorrow, if it's moved out of this package.)

The recommended solution of mixing in every template instance is not a 
viable solution because that would effectively remove IFTI from D. What 
a huge loss that would be. We can't afford that.

So, to be safe, every D code out there that happens to pass a struct to 
a piece of library function would have to

1) Know that that library function happens to be a template and

2) mixin the particular instantiation of that template.

That would be the only sane thing to do. Further, what happens if a 
function changes its implementation and becomes a template that happens 
to call __traits(allMembers)? How can the library notify its users to 
tell them to mixin an explicit instantiation?

Ali
Aug 31 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Wednesday, 31 August 2016 at 21:12:54 UTC, Ali Çehreli wrote:

 On 08/30/2016 03:24 PM, Ali Çehreli wrote:

 v2.071.2-b3 is bringing a change for this bug:

   https://issues.dlang.org/show_bug.cgi?id=15907

 I don't agree with the current solution:

 


 http://dlang.org/changelog/2.071.2.html#traits-members-visibility

 Modules should be able to use library templates without


 needing to mix

 them in first.

 Do you think the solution in the change log usable? I don't


 think so

 because silently skipping my private members is an unexpected


 behavior

 that will cause bugs.


 Here is a regression caused by the above change:

 interface I {
     void foo();

 package:

     void foo(int);
 }

 string how(alias Base, alias func)() {
     return "";
 }

 import std.typecons;
 class C : AutoImplement!(I, how) {
 }


What this derives a class in std.typecons that implements the 
interface, and yes such a derived class can't override/implement 
package protected methods.
AutoImplement would also work better as mixin in order to 
instantiate the passed in how template in the original 
instantiation scope.


 void main() {
     auto c = new C();
     c.foo();
     c.foo(42);    // COMPILATION ERROR:
 // deneme.o: In function `_Dmain':
 // deneme.d:(.text._Dmain+0x39): undefined reference to 
 `_D6deneme1I3fooMFiZv'


The missing interface method implementation should've give a 
compile error, please file a bug.


 The recommended solution of mixing in every template instance 
 is not a viable solution because that would effectively remove 
 IFTI from D. What a huge loss that would be. We can't afford 
 that.


Exaggeration won't help us to find good solutions. Remember that 
private fields never were accessible, so only some edge cases 
will be affected. The need for mixin templates will remain rare.
Sep 03 2016
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/03/2016 09:37 AM, Martin Nowak wrote:

 On Wednesday, 31 August 2016 at 21:12:54 UTC, Ali Çehreli wrote:



 The recommended solution of mixing in every template instance is not a
 viable solution because that would effectively remove IFTI from D.
 What a huge loss that would be. We can't afford that.


 Exaggeration won't help us to find good solutions. Remember that private
 fields never were accessible, so only some edge cases will be affected.
 The need for mixin templates will remain rare.


I don't see how the user of a template library can decide whether to 
mixin or not without knowing the current implementation if of the library.

For that reason, I must mixin. For example, the following program is 
broken because I don't know whether writeln uses allMembers or not:

     // BROKEN CODE:
     MyStruct s;
     writeln(s);

Do you see the problem? I can't leave that code that way. I have to 
figure out mixin writeln's instantiation. As you see, mixins will not be 
rare. Every template use must be mixed in.

Ali
Sep 07 2016
parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 7 September 2016 at 19:29:05 UTC, Ali Çehreli wrote:

 On 09/03/2016 09:37 AM, Martin Nowak wrote:

 On Wednesday, 31 August 2016 at 21:12:54 UTC, Ali Çehreli


 wrote:


 The recommended solution of mixing in every template




 instance is not a

 viable solution because that would effectively remove IFTI




 from D.

 What a huge loss that would be. We can't afford that.


 Exaggeration won't help us to find good solutions. Remember


 that private

 fields never were accessible, so only some edge cases will be


 affected.

 The need for mixin templates will remain rare.


 I don't see how the user of a template library can decide 
 whether to mixin or not without knowing the current 
 implementation if of the library.

 For that reason, I must mixin. For example, the following 
 program is broken because I don't know whether writeln uses 
 allMembers or not:

     // BROKEN CODE:
     MyStruct s;
     writeln(s);

 Do you see the problem? I can't leave that code that way. I 
 have to figure out mixin writeln's instantiation. As you see, 
 mixins will not be rare. Every template use must be mixed in.


Well, I can only repeat what I stated several times before, 
private members were never accessible by name (via getMember or 
.name). So either writeln uses .tupleof (unnamed) or ignores 
private members.

This is the pre 2.071 state.

method    | visible | accessible
--------------------------------
.name     |    y    |    n
getMember |    y    |    n
.tupleof  |    y    |    y

This was the 2.071.1 state (for which the allMembers fix was a 
valid solution).

method    | visible | accessible
--------------------------------
.name     |    n    |    n
getMember |    n    |    n
.tupleof  |    y    |    y

This is what we're now doing in 2.071.2 (see 
https://github.com/dlang/dmd/pull/6111),

method    | visible | accessible
--------------------------------
.name     |    n    |    n
getMember |    y    |    n
.tupleof  |    y    |    y

and what we plan to do with 2.072 or 2.073.

method    | visible | accessible
--------------------------------
.name     |    n    |    n
getMember |    y    |    y
.tupleof  |    y    |    y

In fact access checks will get removed soon.
Sep 12 2016
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I can't wrap my head around the fact that a library template 
 called by my module cannot see my private members.


Well, it was never possible to access them either, didn't seem to 
cause much confusion. Also getSymbolsByUDA is built on a hack 
around that access check 
https://github.com/dlang/phobos/blob/cb09746cb11bcbe7b730f05d29792e6252669175/std/traits.d#L6979.
Note that you can pass private symbols to templates, so 
`hasUDA!(S.invisible, UDA)` works fine.
I don't see any other examples where we'll have much issues with 
that change other than w/ this weird getSymbolsByUDA API, which 
aliases to !(S.field, S.func, S.Nested).
Most templates operating on wholes types already didn't have 
access to private members.
Sep 03 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I don't agree with the current solution:


Well let's come up with a better solution then.
Let's start by finding some proper use-cases that require 
introspection on private members w/o having access to them.
Sep 03 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I don't agree with the current solution:


 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require 
 introspection on private members w/o having access to them.


In my user library, the serialization is based on the  Set and 
 Get UDAs. Typically there would be a setter (a public method) 
and no getter (i.e the data is read directly either from a 
protected or private variable).

The introspection is used to create a property descriptor. But 
because of the protection attributes I have to mix the 
introspection features (== a template) in each aggregate that 
declares properties and in each descendant class that declare new 
properties.



Class Foo
{
     mixin features f; // so that the private stuff are visible

private:
      Get int _field;
public
      Set void field(int value){}

     this()
     {f.analyze!Foo;}
}

class Bar: Foo
{
     mixin features f; // so that the private stuff are visible

private:
      Get int _otherfield;
public
      Set void otherfield(int value){}

     this()
     {f.analyze!Bar;}
}



while I could do, with the omniscient traits:



Class Foo
{
private:
      Get int _field;
public
      Set void field(int value){}

     this(this T)()
     {
         features.analyze!T;
     }
}

Class Bar: Foo
{
private:
      Get int _otherfield;
public
      Set void otherfield(int value){}
}

no need to re-mix the features for each derived class...
Sep 03 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 17:23:53 UTC, Basile B. wrote:

 On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak 
 wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I don't agree with the current solution:


 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require 
 introspection on private members w/o having access to them.


 In my user library, the serialization is based on the  Set and 
  Get UDAs. Typically there would be a setter (a public method) 
 and no getter (i.e the data is read directly either from a 
 protected or private variable).


A public setter for private members is weird, but well. As a 
library template can't read private fields, you don't need the 
 Get attribute.


 The introspection is used to create a property descriptor.


What does that mean? You're creating a property in the serialised 
data?

Defining de-/serialize methods in the class/struct, e.g. with a 
mixin template would be the cleaner and more obvious approach IMO.
Sep 03 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Sunday, 4 September 2016 at 03:45:50 UTC, Martin Nowak wrote:

 On Saturday, 3 September 2016 at 17:23:53 UTC, Basile B. wrote:

 On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak 
 wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I don't agree with the current solution:


 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require 
 introspection on private members w/o having access to them.


 In my user library, the serialization is based on the  Set and 
  Get UDAs. Typically there would be a setter (a public method) 
 and no getter (i.e the data is read directly either from a 
 protected or private variable).


 A public setter for private members is weird, but well.


No in Object Pascal that's a common to have

     property Foo: string read fFoo write setFoo;

By the way there's an error from my part, in my lib it's  SetGet 
which gives access to private/protected field "directly".


 As a library template can't read private fields, you don't need 
 the  Get attribute.


 The introspection is used to create a property descriptor.


 What does that mean? You're creating a property in the 
 serialised data?


The introspection creates a special structure for each property 
annotated with  Set,  Get, or  SetGet. This is a kind of 
interface for serialization/binding/object inspector (a bit like 
"published" in Object Pascal).

https://gist.github.com/BBasile/39fb66f7a0189660182cc637ab8d698b/archive/3e0f7441cfba89b5959b78ee
63b9538d137a55a.zip (you can "dub a.d")


 Defining de-/serialize methods in the class/struct, e.g. with a 
 mixin template would be the cleaner and more obvious approach 
 IMO.


I see your strategy...each time someone will find an argument to 
make the traits omniscients you'll say, "no because it's cleaner 
to do like that" ?
Sep 04 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 4 September 2016 at 07:24:42 UTC, Basile B. wrote:

 The introspection creates a special structure for each property 
 annotated with  Set,  Get, or  SetGet. This is a kind of 
 interface for serialization/binding/object inspector (a bit 
 like "published" in Object Pascal).


But you can't access private fields outside of the current 
module, so you need to mixin sth. to set/get the value anyhow.


 Defining de-/serialize methods in the class/struct, e.g. with 
 a mixin template would be the cleaner and more obvious 
 approach IMO.


 I see your strategy...each time someone will find an argument 
 to make the traits omniscients you'll say, "no because it's 
 cleaner to do like that" ?


No, I'm just trying to understand the requirement for 
introspection without access.
Sep 04 2016
parent Basile B. <b2.temp gmx.com> writes:
On Sunday, 4 September 2016 at 08:42:43 UTC, Martin Nowak wrote:

 On Sunday, 4 September 2016 at 07:24:42 UTC, Basile B. wrote:

 The introspection creates a special structure for each 
 property annotated with  Set,  Get, or  SetGet. This is a kind 
 of interface for serialization/binding/object inspector (a bit 
 like "published" in Object Pascal).


 But you can't access private fields outside of the current 
 module, so you need to mixin sth. to set/get the value anyhow.


But that's the point ! We are many willing to see this 
restriction removed for the traits because they are special., i.e 
they should allow to implement, using meta programming, features 
that less powerful languages do in intern. That's also why I 
propose to add a 3rd optional bool parameter to the allMembers 
trait:

enum RespectProtection = false;
enum IgnoreProtection = true;

__traits(allMembers, T, RespectProtection)
__traits(allMembers, T, IgnoreProtection)

The 3rd optional parameter, when not specified, would be set a 
default value of false (equivalent to __traits(allMembers, T, 
RespectProtection), to prevent breakage.

Because allMembers would act as a filter for getMember, 
getOverload...the protection check in getMember, getOverload... 
would be removed.

This will be obviously more complex than that but that's the idea.
Sep 04 2016
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-09-04 05:45, Martin Nowak wrote:


 A public setter for private members is weird, but well.


The setter can do validations, manipulation and other kinds of 
calculations based on the given value.

-- 
/Jacob Carlborg
Sep 04 2016
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 5:45 AM, Martin Nowak wrote:

 A public setter for private members is weird


How do you mean that? It's an absolute classic. Please elaborate, 
thanks. -- Andrei
Sep 04 2016
prev sibling next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Usability of "allMembers and derivedMembers traits now only
 return visible symbols"
References: <nq512d$9po$1 digitalmars.com>
 <ghusqqkmtakmehzyxpsl forum.dlang.org>
In-Reply-To: <ghusqqkmtakmehzyxpsl forum.dlang.org>

--qarTW6Qv25f05gfq2sSivoM3CvOUUC4e2
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 09/03/2016 07:52 PM, Martin Nowak wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali =C3=87ehreli wrote:

 I don't agree with the current solution:


=20
 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require introspection=



 on private members w/o having access to them.


Obviously serialization libraries come to mind, especially important for
binary serialization because private members still affect the layout of
the struct.


--qarTW6Qv25f05gfq2sSivoM3CvOUUC4e2--
Sep 03 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-03 20:56, Dicebot wrote:


 Obviously serialization libraries come to mind, especially important for
 binary serialization because private members still affect the layout of
 the struct.


That can usually be solved using .tupleof[i], but I agree.

-- 
/Jacob Carlborg
Sep 03 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 19:31:12 UTC, Jacob Carlborg 
wrote:

 That can usually be solved using .tupleof[i], but I agree.


Well, .tupleof gives you a typed representation of the memory 
layout, to me it's something different than qualified access of 
fields, just a safer mean to access memory. And sure you can 
always memcpy data.
Also introspection works on .tupleof, so it's not affected by the 
allMembers change and it's awkward to mix them b/c names and 
layout don't have a simple 1-to-1 relation.
Sep 03 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 5:56 AM, Martin Nowak wrote:

 On Saturday, 3 September 2016 at 19:31:12 UTC, Jacob Carlborg wrote:

 That can usually be solved using .tupleof[i], but I agree.


 Well, .tupleof gives you a typed representation of the memory layout, to
 me it's something different than qualified access of fields, just a
 safer mean to access memory. And sure you can always memcpy data.
 Also introspection works on .tupleof, so it's not affected by the
 allMembers change and it's awkward to mix them b/c names and layout
 don't have a simple 1-to-1 relation.


Yah, .tupleof is great to have. I think that should be enough for most 
introspection needs. Only the field names are missing, can those be 
accessed somehow? -- Andrei
Sep 04 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-04 14:36, Andrei Alexandrescu wrote:


 Yah, .tupleof is great to have. I think that should be enough for most
 introspection needs. Only the field names are missing, can those be
 accessed somehow? -- Andrei


Yes:

module foo;

struct Foo
{
     private int a;
}

module main;

import foo;
static assert(__traits(identifier, Foo.tupleof[0]) == "a");

-- 
/Jacob Carlborg
Sep 05 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/5/16 10:38 AM, Jacob Carlborg wrote:

 On 2016-09-04 14:36, Andrei Alexandrescu wrote:


 Yah, .tupleof is great to have. I think that should be enough for most
 introspection needs. Only the field names are missing, can those be
 accessed somehow? -- Andrei


 Yes:

 module foo;

 struct Foo
 {
     private int a;
 }

 module main;

 import foo;
 static assert(__traits(identifier, Foo.tupleof[0]) == "a");


Thank you. This is good news. -- Andrei
Sep 05 2016
prev sibling parent reply Johan Engelen <j j.nl> writes:
On Sunday, 4 September 2016 at 12:36:43 UTC, Andrei Alexandrescu 
wrote:

 Yah, .tupleof is great to have. I think that should be enough 
 for most introspection needs. Only the field names are missing, 
 can those be accessed somehow? -- Andrei


Note that the spec tells us:
"The .tupleof property returns an ExpressionTuple of all the 
fields in the class, __excluding__ the hidden fields and __the 
fields in the base class__." (emphasis mine)
So then you have to do __traits(parent, ) recursively...?
Sep 07 2016
parent Basile B. <b2.temp gmx.com> writes:
On Wednesday, 7 September 2016 at 15:08:17 UTC, Johan Engelen 
wrote:

 Note that the spec tells us:
 "The .tupleof property returns an ExpressionTuple of all the 
 fields in the class, __excluding__ the hidden fields and __the 
 fields in the base class__." (emphasis mine)
 So then you have to do __traits(parent, ) recursively...?


or use std.traits.BaseClassesTuple.
Sep 07 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 18:56:33 UTC, Dicebot wrote:

 Obviously serialization libraries come to mind, especially 
 important for binary serialization because private members 
 still affect the layout of the struct.


Let me repeat that another time, private members have never been 
accessible by traits. That's the main reasoning this change was 
based upon.
Sep 03 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require 
 introspection on private members w/o having access to them.


Still haven't heard really good use cases for the above, but the 
trade-offs for also allowing access to private members aren't 
that bad, thus it seems like the better path forward.
See 
http://forum.dlang.org/post/ymkehalxcigswvltlfjj forum.dlang.org
Sep 04 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/4/16 11:09 AM, Martin Nowak wrote:

 On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:
 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require
 introspection on private members w/o having access to them.


 Still haven't heard really good use cases for the above, but the
 trade-offs for also allowing access to private members aren't that bad,
 thus it seems like the better path forward.
 See http://forum.dlang.org/post/ymkehalxcigswvltlfjj forum.dlang.org


There are a few cases I can think of:

* Serialization and deserialization, shallow and deep. These would need 
access to the object layout, and possibly field names too (for 
cross-checking and versioning purposes).

* Binary saving, loading, and fixup (a subset of 
serialization/deserialization) - that thing when you copy raw memory to 
a file and then load it raw and fix pointers up.

* Memory management and garbage collection: access to field types allows 
efficient generic tracing.

* Database interfacing, automated binding, object-relational mapping, 
etc. It stands to reason that field names would be needed and fields 
would be private, yet the database loader does have access to them.

Of course the more important applications are those I can't yet imagine. 
The way I see it introspection must have full power and unfettered 
access. So definitely it must be able to pass through regular protection.


Andrei
Sep 04 2016
next sibling parent Martin Nowak <code dawg.eu> writes:
On Sunday, 4 September 2016 at 12:51:05 UTC, Andrei Alexandrescu 
wrote:

 See 
 http://forum.dlang.org/post/ymkehalxcigswvltlfjj forum.dlang.org


 There are a few cases I can think of:


The classical ones, but again the question was for introspection 
without access (b/c named access through getMember was never 
allowed, only .tupleof).
As the trade-offs for allowing access to private members don't 
seem that bad, we'll now try to go in the other direction, 
skipping visibility checks now and removing the access checks in 
a later release.
Sep 04 2016
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-09-04 14:51, Andrei Alexandrescu wrote:


 Of course the more important applications are those I can't yet imagine.
 The way I see it introspection must have full power and unfettered
 access. So definitely it must be able to pass through regular protection.


* Linking up GUI controls to instance variables in the code, like 
Xcode/Interface Builder

-- 
/Jacob Carlborg
Sep 05 2016
prev sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/04/2016 05:51 AM, Andrei Alexandrescu wrote:

* Tracing how and when a member mutates.

Ali
Sep 06 2016
prev sibling parent earthfront <earthfront mailinator.com> writes:
On Saturday, 3 September 2016 at 16:52:50 UTC, Martin Nowak wrote:

 On Tuesday, 30 August 2016 at 22:24:12 UTC, Ali Çehreli wrote:

 I don't agree with the current solution:


 Well let's come up with a better solution then.
 Let's start by finding some proper use-cases that require 
 introspection on private members w/o having access to them.


Use case that I think is pretty durn important: using 
Allocator.make and forwarding a call to the private constructor 
on a class.

This does not work now:
---
class A
{ int b; private this(int a){b=a;} }

   auto ptr = make!A(Mallocator.instance, 42); // Compile Error!
---

Allocators are second class in this respect compared to GC.

Mixins aren't a good solution to this.
Sep 14 2016