www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Problem with coupling shared object symbol visibility with protection

reply Benjamin Thaut <code benjamin-thaut.de> writes:
I'm currently working on Windows DLL support which has stronger rules 
than linux shared objects for which symbols actually get exported from a 
shared library. But as we want to replicate the same behaviour on linux 
using symbol visibility (e.g. gcc 4 -fVisibility=hidden) this issue also 
applies to linux once implemented.

Currently export means two things:
- the symbol is publicy accessible (same as public)
- the symbol will be accisble across shared library boundaries


This has the following issue:

export void templateFunc(T)()
{
   someHelperFunc();
}

private void someHelperFunc()
{

}

And you don't even have to go into phobos to hit this problem. It is 
already in druntime see core.time.FracSec._enforceValid

This works with the current linux shared objects because they simply 
export all symbols. But once only symbols with export get exported this 
breaks.

The problem here is that you don't want to make someHelperFunc() export 
because that would mean users could call it directly, but you want it to 
be available for cross shared library calls. The cross shared library 
call happens if a template is instanced from a different shared library 
/ executable than the module it was originally located in.

There are two solutions for this.

1) Given that export is transitive (that means if a struct or class is 
declared export every member that is _not_ private will be accessible 
across shared library boundaries. This behaviour is required to make the 
export protection level work on windows)

You can now do the following:

export struct SomeStruct
{
   static public void templateFunc(T)()
   {
     someHelperFunc();
   }

   static package void someHelperFunc()
   {

   }
}

Because of the transitivity someHelperFunc will be exported but still 
not be callable by the user directly. This can be used to fix the issue 
in core.time but may not be so well suited if you want the template to 
be on module level instead of nesting it into a struct.

2) Make export an attribute. If export is no longer an protection level 
but instead an attribute this issue can easily be solved by doing.

export public void templateFunc(T)()
{
   someHelperFunc();
}

export private void someHelperFunc()
{

}

But this would require grammar changes. Which are generally avoided if 
possible.

There would be a third option, which I rather avoid. Doing a 
"pramga(forceExport)" or something like that.

My implementation, which I ran into this issue with, currently usses 
approach 1. What do you think how this sould be solved?

Walter: What was the general idea behind export when you designed it, 
and how can it be used to solve this problem?

Kind Regards
Benjamin Thaut
Jan 20 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
I like the first version more as it fits better the natural way 
people design their interfaces and reduces attributed noise. 
Can't say if it will trigger some hidden issues.

First version also fits better some of ideas I had about 
automated API generation/verification 
(http://forum.dlang.org/post/otejdbgnhmyvbyaxatsk forum.dlang.org)

Thanks for keeping to poke this issue - symbol visibility is 
currently a big undefined minefield in D ABI.
Jan 21 2015
parent "Benjamin Thaut" <code benjamin-thaut.de> writes:
 Thanks for keeping to poke this issue - symbol visibility is 
 currently a big undefined minefield in D ABI.


Your welcome. At this point I'm so desperate for D Dll support 
that I stopped poking and started implementing it myself. I'm 3 
unresolved symbol references away from actually building phobos 
into a dll (druntime already is).

By the way you could also export non accessible symbols like this:

static public void templateFunc(T)()
{
   Impl.someHelperFunc();
}

export struct Impl
{
   package:
   static void someHelperFunc()
   {

   }
}
Jan 21 2015
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 20 January 2015 at 12:23:32 UTC, Benjamin Thaut wrote:

 I'm currently working on Windows DLL support which has stronger 
 rules than linux shared objects for which symbols actually get 
 exported from a shared library. But as we want to replicate the 
 same behaviour on linux using symbol visibility (e.g. gcc 4 
 -fVisibility=hidden) this issue also applies to linux once 
 implemented.

 Currently export means two things:
 - the symbol is publicy accessible (same as public)
 - the symbol will be accisble across shared library boundaries


 This has the following issue:

 export void templateFunc(T)()
 {
   someHelperFunc();
 }

 private void someHelperFunc()
 {

 }

 And you don't even have to go into phobos to hit this problem. 
 It is already in druntime see core.time.FracSec._enforceValid

 This works with the current linux shared objects because they 
 simply export all symbols. But once only symbols with export 
 get exported this breaks.

 The problem here is that you don't want to make 
 someHelperFunc() export because that would mean users could 
 call it directly, but you want it to be available for cross 
 shared library calls. The cross shared library call happens if 
 a template is instanced from a different shared library / 
 executable than the module it was originally located in.

 There are two solutions for this.

 1) Given that export is transitive (that means if a struct or 
 class is declared export every member that is _not_ private 
 will be accessible across shared library boundaries. This 
 behaviour is required to make the export protection level work 
 on windows)

 You can now do the following:

 export struct SomeStruct
 {
   static public void templateFunc(T)()
   {
     someHelperFunc();
   }

   static package void someHelperFunc()
   {

   }
 }

 Because of the transitivity someHelperFunc will be exported but 
 still not be callable by the user directly. This can be used to 
 fix the issue in core.time but may not be so well suited if you 
 want the template to be on module level instead of nesting it 
 into a struct.

 2) Make export an attribute. If export is no longer an 
 protection level but instead an attribute this issue can easily 
 be solved by doing.

 export public void templateFunc(T)()
 {
   someHelperFunc();
 }

 export private void someHelperFunc()
 {

 }

 But this would require grammar changes. Which are generally 
 avoided if possible.

 There would be a third option, which I rather avoid. Doing a 
 "pramga(forceExport)" or something like that.

 My implementation, which I ran into this issue with, currently 
 usses approach 1. What do you think how this sould be solved?

 Walter: What was the general idea behind export when you 
 designed it, and how can it be used to solve this problem?

 Kind Regards
 Benjamin Thaut


Just as heads up in case D ever comes to Aix, I don't know how it 
looks like nowadays, but Aix back in 2000 used to be have similar 
behavior to Windows.

The .def files in Windows were .exp (I think) files on Aix.

--
Paulo
Jan 21 2015
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
There are uses in Phobos where workaround 1) would require some code 
changes:

private  property File trustedStdout()  trusted { return stdout; }

void write(T...)(T args) if (!is(T[0] : File))
{
     trustedStdout.write(args);
}

My workaround so far:

export struct _impl1
{
   package  property static File trustedStdout()  trusted { return stdout; }
}

alias trustedStdout = _impl1.trustedStdout;
Jan 22 2015
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/20/2015 4:23 AM, Benjamin Thaut wrote:

 I'm currently working on Windows DLL support which has stronger rules than
linux
 shared objects for which symbols actually get exported from a shared library.
 But as we want to replicate the same behaviour on linux using symbol visibility
 (e.g. gcc 4 -fVisibility=hidden) this issue also applies to linux once
implemented.

 Currently export means two things:
 - the symbol is publicy accessible (same as public)
 - the symbol will be accisble across shared library boundaries


 This has the following issue:

 export void templateFunc(T)()
 {
    someHelperFunc();
 }

 private void someHelperFunc()
 {

 }

 And you don't even have to go into phobos to hit this problem. It is already in
 druntime see core.time.FracSec._enforceValid

 This works with the current linux shared objects because they simply export all
 symbols. But once only symbols with export get exported this breaks.

 The problem here is that you don't want to make someHelperFunc() export because
 that would mean users could call it directly, but you want it to be available
 for cross shared library calls. The cross shared library call happens if a
 template is instanced from a different shared library / executable than the
 module it was originally located in.


exporting a template and then having the user instantiate outside of the
library 
doesn't make a whole lot of sense, because the instantiation won't be there in 
the library. The library will have to instantiate every use case. If the 
compiler knows the library instantiated it, it won't re-instantiate it locally.



 There are two solutions for this.

 1) Given that export is transitive (that means if a struct or class is declared
 export every member that is _not_ private will be accessible across shared
 library boundaries. This behaviour is required to make the export protection
 level work on windows)

 You can now do the following:

 export struct SomeStruct
 {
    static public void templateFunc(T)()
    {
      someHelperFunc();
    }

    static package void someHelperFunc()
    {

    }
 }

 Because of the transitivity someHelperFunc will be exported but still not be
 callable by the user directly. This can be used to fix the issue in core.time
 but may not be so well suited if you want the template to be on module level
 instead of nesting it into a struct.

 2) Make export an attribute. If export is no longer an protection level but
 instead an attribute this issue can easily be solved by doing.

 export public void templateFunc(T)()
 {
    someHelperFunc();
 }

 export private void someHelperFunc()
 {

 }

 But this would require grammar changes. Which are generally avoided if
possible.

 There would be a third option, which I rather avoid. Doing a
 "pramga(forceExport)" or something like that.

 My implementation, which I ran into this issue with, currently usses approach
1.
 What do you think how this sould be solved?


I'd be thinking that what a shared library exports is fixed, and expecting the 
user to make more instantiations makes for a fairly unresolvable issue. The 
solution is design the templates to be expanded only on one side of the dll/exe 
boundary, not straddle it.
Jan 26 2015
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 26.01.2015 um 23:24 schrieb Walter Bright:

 exporting a template and then having the user instantiate outside of the
 library doesn't make a whole lot of sense, because the instantiation
 won't be there in the library. The library will have to instantiate
 every use case. If the compiler knows the library instantiated it, it
 won't re-instantiate it locally.


Sorry, but wtf? So we just throw all of phobos away? Given this argument 
you can _never_ make phobos into a shared lirary becuse you can't 
possibliy pre-instaniate all possible template permutations in phobos. 
Your suggestion is completely un-pratical. Even C++ allows you to 
instanciate templates provided by a shared library.


 I'd be thinking that what a shared library exports is fixed, and
 expecting the user to make more instantiations makes for a fairly
 unresolvable issue. The solution is design the templates to be expanded
 only on one side of the dll/exe boundary, not straddle it.


Again this just makes it impossible to make phobos ever work as a shared 
library. A Language as template heavy as D should clearly allow users to 
instanciate templates across dll/exe boundaries.

Why am I getting the impression, everytime I read one of your comments 
regarding dlls on windows, that you don't want them to be useable? DO 
you have a special hate against Windows or their shared library solution?

Kind Regards
Benjamin Thaut
Jan 26 2015
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 26.01.2015 23:24, Walter Bright wrote:

 The problem here is that you don't want to make someHelperFunc()
 export because that would mean users could call it directly, but
 you want it to be available for cross shared library calls. The
 cross shared library call happens if a template is instanced from a
 different shared library / executable than the module it was
 originally located in.


 exporting a template and then having the user instantiate outside of
 the library doesn't make a whole lot of sense, because the
 instantiation won't be there in the library. The library will have to
 instantiate every use case. If the compiler knows the library
 instantiated it, it won't re-instantiate it locally.


The problem is not about into which binary the template is generated to 
(this must be the binary where it is used), but how to access private 
non-templated methods called by the template.

 From the core.time.FracSec example:

export struct FracSec
{
     ///...
     static FracSec from(string units)(long value)
         if(units == "msecs" ||
            units == "usecs" ||
            units == "hnsecs" ||
            units == "nsecs")
     {
         immutable hnsecs = cast(int)convert!(units, "hnsecs")(value);
         _enforceValid(hnsecs);
         return FracSec(hnsecs);
     }

     private static void _enforceValid(int hnsecs)
     {
         if(!_valid(hnsecs))
             throw new TimeException("FracSec must ...");
     }
     ///...
}

_enforceValid() could also be a free function. It is likely to be 
compiled into druntime.dll, but needs to be exported from the DLL to be 
callable by the instantiation of the template function in another DLL. 
The "private" forbids exporting, though.
Jan 27 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/27/2015 2:05 PM, Rainer Schuetze wrote:

 On 26.01.2015 23:24, Walter Bright wrote:

 The problem here is that you don't want to make someHelperFunc()
 export because that would mean users could call it directly, but
 you want it to be available for cross shared library calls. The
 cross shared library call happens if a template is instanced from a
 different shared library / executable than the module it was
 originally located in.


 exporting a template and then having the user instantiate outside of
 the library doesn't make a whole lot of sense, because the
 instantiation won't be there in the library. The library will have to
 instantiate every use case. If the compiler knows the library
 instantiated it, it won't re-instantiate it locally.


 The problem is not about into which binary the template is generated to (this
 must be the binary where it is used),


The example had marked the template itself as 'export'. This raises the specter 
of which binary the template instantiation exists in.


 but how to access private non-templated methods called by the template.

  From the core.time.FracSec example:

 export struct FracSec
 {
      ///...
      static FracSec from(string units)(long value)
          if(units == "msecs" ||
             units == "usecs" ||
             units == "hnsecs" ||
             units == "nsecs")
      {
          immutable hnsecs = cast(int)convert!(units, "hnsecs")(value);
          _enforceValid(hnsecs);
          return FracSec(hnsecs);
      }

      private static void _enforceValid(int hnsecs)
      {
          if(!_valid(hnsecs))
              throw new TimeException("FracSec must ...");
      }
      ///...
 }

 _enforceValid() could also be a free function. It is likely to be compiled into
 druntime.dll, but needs to be exported from the DLL to be callable by the
 instantiation of the template function in another DLL. The "private" forbids
 exporting, though.


I tend to view a DLL's exports as being inherently not private, hence D's
export 
design being a "super" public. At the risk of sounding flip, I suggest simply 
removing the 'private' from free functions one wishes to export. If the user is 
calling undocumented functions that start with '_', we can presume they know 
what they're doing.

If you still want to hide the free function, it can be done like this:

   struct Bar(T) {
     void callit() { Impl.freefunc(); }
   }

   private struct Impl {
         export static void freefunc() { }
   }
Jan 28 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
My 5 cents:

1) "export all" approach is extremely limiting and I'd like to 
see it go on Linux too. One of major problems with it is that 
many forms of link-time optimizations (such as --gc-sections) 
become simply impossible when compiler can't know what symbols 
are actually supposed to be available externally.

2) first proposed solution does not allow to mark private 
functions as "export". It implicitly exports those if they are 
needed for actual public/export template function to work. This 
is not the same - those functions still can't be called via 
provide .di binding, it simply keeps them available for linking.

3) there is a big maintenance benefit from encouraging people to 
explicitly mark their API, template or not. It is a clear sign 
"this is a part of my libraries you are expected to use directly" 
and that was what my idea for attribute inference abused.
Jan 28 2015
parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Wednesday, 28 January 2015 at 11:42:19 UTC, Dicebot wrote:

 2) first proposed solution does not allow to mark private 
 functions as "export". It implicitly exports those if they are 
 needed for actual public/export template function to work. This 
 is not the same - those functions still can't be called via 
 provide .di binding, it simply keeps them available for linking.


So you would prefer to keep symbols private and just make them 
available for linking? Or what exactly is the message here?


 3) there is a big maintenance benefit from encouraging people 
 to explicitly mark their API, template or not. It is a clear 
 sign "this is a part of my libraries you are expected to use 
 directly" and that was what my idea for attribute inference 
 abused.


Agree here. Especially if you want to keep your shared library 
backwards compatible, so that people can simply drop in the new 
binary. To do that you need very fine control over what is 
exported and what is not.
Jan 28 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 28 January 2015 at 13:30:17 UTC, Benjamin Thaut 
wrote:

 On Wednesday, 28 January 2015 at 11:42:19 UTC, Dicebot wrote:

 2) first proposed solution does not allow to mark private 
 functions as "export". It implicitly exports those if they are 
 needed for actual public/export template function to work. 
 This is not the same - those functions still can't be called 
 via provide .di binding, it simply keeps them available for 
 linking.


 So you would prefer to keep symbols private and just make them 
 available for linking? Or what exactly is the message here?


Isn't that what your first proposed solution is about? That was 
my understanding and I liked that understanding :) To be 100% 
clear :

export void foo(T : int)(T x)
{
     bar(x);
}

private void bar(int x) { }

I'd expect `bar` to be exported into resulting binary so that it 
can be linked against by any users of `foo` but for any attempt 
to call `bar` directly from D code to fail because of protection 
violation. If someone wants to circumvent protection by forging 
mangling - shooting own feet is allowed.
Jan 28 2015
next sibling parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Wednesday, 28 January 2015 at 13:48:45 UTC, Dicebot wrote:

 Isn't that what your first proposed solution is about? That was 
 my understanding and I liked that understanding :) To be 100% 
 clear :

 export void foo(T : int)(T x)
 {
     bar(x);
 }

 private void bar(int x) { }

 I'd expect `bar` to be exported into resulting binary so that 
 it can be linked against by any users of `foo` but for any 
 attempt to call `bar` directly from D code to fail because of 
 protection violation. If someone wants to circumvent protection 
 by forging mangling - shooting own feet is allowed.


Well this would be ultimate goal. Although it is not possible to 
automatically detect that bar needs to be exported because that 
would mean you would have to analyze all possible instantiations 
of the template foo. (static if...)
So Automatically detecting that bar needs to be exported is not 
possible. We either have to make export into an attribute or use 
the solution where bar is actually nested into a struct which is 
exported instead. This is also described in my initial post with 
examples.
Jan 28 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 28 January 2015 at 14:16:03 UTC, Benjamin Thaut 
wrote:

 Well this would be ultimate goal. Although it is not possible 
 to automatically detect that bar needs to be exported because 
 that would mean you would have to analyze all possible 
 instantiations of the template foo. (static if...)
 So Automatically detecting that bar needs to be exported is not 
 possible. We either have to make export into an attribute or 
 use the solution where bar is actually nested into a struct 
 which is exported instead. This is also described in my initial 
 post with examples.


Yes, I see the problem now. "static if" isn't even the worst 
offender, any kind of string mixins that generate the call to 
`bar` are impossible to detect without having actual `foo` 
instance. Sorry for misinterpretation.

With that in mind second option is starting to look more 
attractive despite the grammar change - forcing good chunk 
template API into structs does not sound very convenient :(
Jan 28 2015
parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Wednesday, 28 January 2015 at 14:31:54 UTC, Dicebot wrote:


 Yes, I see the problem now. "static if" isn't even the worst 
 offender, any kind of string mixins that generate the call to 
 `bar` are impossible to detect without having actual `foo` 
 instance. Sorry for misinterpretation.

 With that in mind second option is starting to look more 
 attractive despite the grammar change - forcing good chunk 
 template API into structs does not sound very convenient :(


Well and then there is the third option Walter proposed:
- Make everything export even if it means that it gets callable 
by the user directly.
Jan 28 2015
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 28 January 2015 at 15:01:30 UTC, Benjamin Thaut 
wrote:

 With that in mind second option is starting to look more 
 attractive despite the grammar change - forcing good chunk 
 template API into structs does not sound very convenient :(


 Well and then there is the third option Walter proposed:
 - Make everything export even if it means that it gets callable 
 by the user directly.


It is hardly a good option - I consider LTO/WPO a very important 
future goal for any native language.
Jan 28 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 28 January 2015 at 13:48:45 UTC, Dicebot wrote:

 Isn't that what your first proposed solution is about? That was 
 my understanding and I liked that understanding :) To be 100% 
 clear :

 export void foo(T : int)(T x)
 {
     bar(x);
 }

 private void bar(int x) { }

 I'd expect `bar` to be exported into resulting binary so that 
 it can be linked against by any users of `foo` but for any 
 attempt to call `bar` directly from D code to fail because of 
 protection violation. If someone wants to circumvent protection 
 by forging mangling - shooting own feet is allowed.


So you'd want bar to be duplicated on both sides ? This is gonna 
cause problems with di files.
Jan 31 2015
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 31.01.2015 um 23:42 schrieb deadalnix:

 On Wednesday, 28 January 2015 at 13:48:45 UTC, Dicebot wrote:

 Isn't that what your first proposed solution is about? That was my
 understanding and I liked that understanding :) To be 100% clear :

 export void foo(T : int)(T x)
 {
     bar(x);
 }

 private void bar(int x) { }

 I'd expect `bar` to be exported into resulting binary so that it can
 be linked against by any users of `foo` but for any attempt to call
 `bar` directly from D code to fail because of protection violation. If
 someone wants to circumvent protection by forging mangling - shooting
 own feet is allowed.


 So you'd want bar to be duplicated on both sides ? This is gonna cause
 problems with di files.


No. He wants the compiler to automatically detect that the template foo 
might call bar. As a result the compiler should export bar when 
compiling the shared library so that any user of foo does not run into a 
"unresolved symbol reference" linker error. Bar would still only exist 
once: within the shared library.
Feb 01 2015
parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 1 February 2015 at 09:24:46 UTC, Benjamin Thaut wrote:

 So you'd want bar to be duplicated on both sides ? This is 
 gonna cause
 problems with di files.


 No. He wants the compiler to automatically detect that the 
 template foo might call bar. As a result the compiler should 
 export bar when compiling the shared library so that any user 
 of foo does not run into a "unresolved symbol reference" linker 
 error. Bar would still only exist once: within the shared 
 library.


Correct. Sadly it won't work that way as I was already pointed at.
Otherwise it is perfect solution :P
Feb 01 2015
prev sibling next sibling parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright 
wrote:


 The example had marked the template itself as 'export'. This 
 raises the specter of which binary the template instantiation 
 exists in.


The export in this context actually means "export all 
instanciations of this template". And this is needed to avoid 
using -allinst everywhere.
Jan 28 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/28/2015 5:19 AM, Benjamin Thaut wrote:

 On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright wrote:


 The example had marked the template itself as 'export'. This raises the
 specter of which binary the template instantiation exists in.


 The export in this context actually means "export all instanciations of this
 template". And this is needed to avoid using -allinst everywhere.


The problem is what happens when the client side instantiates the template, 
which it must in order to use it.
Jan 29 2015
parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Thursday, 29 January 2015 at 10:21:25 UTC, Walter Bright wrote:

 On 1/28/2015 5:19 AM, Benjamin Thaut wrote:

 On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright 
 wrote:


 The example had marked the template itself as 'export'. This 
 raises the
 specter of which binary the template instantiation exists in.


 The export in this context actually means "export all 
 instanciations of this
 template". And this is needed to avoid using -allinst 
 everywhere.


 The problem is what happens when the client side instantiates 
 the template, which it must in order to use it.


Well if there already is a statically known instanciation it will 
not instanciate it. (I didn't change that behvaior). The question 
is what happens when you have something like this:

module a:
struct SomeTemplate(T){}
alias knownInstance = SomeTemplate!int;

module b:
SomeTemplate!int var1; // will use instanciation from a (unless 
-allinst)
SomeTemplate!float var2; // will instanciate
alias knownInstance2 = SomeTemplate!uint;

module c:
SomeTemplate!uint var3; // will this use instaction from b? Or 
instanciate itself?

I don't know enough about D's template implementation to answer 
the question regarding c.var3. Depending on the answer to this 
question I can answer what should happen if a export marked 
template is instanciated outside of its module. (e.g. by the user)

Please also correct me if any of the above assumptions are 
incorrect.
Jan 29 2015
parent reply "Martin Nowak" <code dawg.eu> writes:
On Thursday, 29 January 2015 at 13:24:36 UTC, Benjamin Thaut 
wrote:

 module c:
 SomeTemplate!uint var3; // will this use instaction from b? Or 
 instanciate itself?


That's the first instantiation with uint. If you mean float, then 
it will instatiate the template when compiled individually and 
use b's instantiation when b and c are compiled together.
Jan 30 2015
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.01.2015 um 11:39 schrieb Martin Nowak:

 If you mean float, then it
 will instatiate the template when compiled individually and use b's
 instantiation when b and c are compiled together.


If this is true, then please explain this behavior:

module a; // --> a.dll

struct Storage(T)
{
   T var;
   __gshared T s_var;
}


module b; // --> b.dll
import a;

export __gshared Storage!int g_var1;


module c; // --> c.exe
import a;
import b;

__gshared Storage!int g_var2;

void main(string[] args)
{
   g_var1.var = 2;
   g_var2.var = 3;

   g_var1.s_var = 2;
   g_var2.s_var = 3;
}

c.obj : error LNK2019: Verweis auf nicht aufgelöstes externes Symbol 
"_D1a14__T7StorageTiZ7Storage5s_vari" in Funktion "_Dmain".


If your statement would be true module c should not use the instance 
from module b because they are not compiled together. But as you can 
clearly see from the linker error message module c does not instanciate 
the template, because if module c would instanciate the template there 
would not be a unresolved symbol.

Compiling module c with "-allinst" solves the problem. Putting "export" 
in front of struct Storage(T) also solves the problem. And thats exactly 
the reason why it is neccessary to be able to export templates. (Where 
export means, export all template instanciations)
Jan 30 2015
prev sibling parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Wednesday, 28 January 2015 at 11:01:09 UTC, Walter Bright 
wrote:

 The example had marked the template itself as 'export'. This 
 raises the specter of which binary the template instantiation 
 exists in.


Also sorry for the harsh answer, this was a classical double 
misunderstanding.
Jan 28 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/28/2015 5:27 AM, Benjamin Thaut wrote:

 Also sorry for the harsh answer, this was a classical double misunderstanding.


No problemo.
Jan 29 2015
prev sibling next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 20.01.2015 13:23, Benjamin Thaut wrote:

 I'm currently working on Windows DLL support which has stronger rules
 than linux shared objects for which symbols actually get exported from a
 shared library. But as we want to replicate the same behaviour on linux
 using symbol visibility (e.g. gcc 4 -fVisibility=hidden) this issue also
 applies to linux once implemented.

 Currently export means two things:
 - the symbol is publicy accessible (same as public)
 - the symbol will be accisble across shared library boundaries


 This has the following issue:

 export void templateFunc(T)()
 {
    someHelperFunc();
 }

 private void someHelperFunc()
 {

 }

 And you don't even have to go into phobos to hit this problem. It is
 already in druntime see core.time.FracSec._enforceValid

 This works with the current linux shared objects because they simply
 export all symbols. But once only symbols with export get exported this
 breaks.


I would not mind if we export all symbols on Windows aswell. It doesn't 
seem to bother a lot of people for the linux version, even though it's 
unsafer and slower than on Windows (at least that was my experience more 
than 10 years ago). It might get us a first working version without 
adding "export" throughout the druntime/phobos source code.


 The problem here is that you don't want to make someHelperFunc() export
 because that would mean users could call it directly, but you want it to
 be available for cross shared library calls. The cross shared library
 call happens if a template is instanced from a different shared library
 / executable than the module it was originally located in.

 There are two solutions for this.

 1) Given that export is transitive (that means if a struct or class is
 declared export every member that is _not_ private will be accessible
 across shared library boundaries. This behaviour is required to make the
 export protection level work on windows)

 You can now do the following:

 export struct SomeStruct
 {
    static public void templateFunc(T)()
    {
      someHelperFunc();
    }

    static package void someHelperFunc()
    {

    }
 }

 Because of the transitivity someHelperFunc will be exported but still
 not be callable by the user directly. This can be used to fix the issue
 in core.time but may not be so well suited if you want the template to
 be on module level instead of nesting it into a struct.

 2) Make export an attribute. If export is no longer an protection level
 but instead an attribute this issue can easily be solved by doing.

 export public void templateFunc(T)()
 {
    someHelperFunc();
 }

 export private void someHelperFunc()
 {

 }

 But this would require grammar changes. Which are generally avoided if
 possible.


I don't have a clear favorite, but the second version makes it clearer 
that visibility and protection are separate issues.

A note on:

 export public void templateFunc(T)()


I don't think it is well defined what exporting a template is supposed 
to mean. My guess: whenever an instance of the template is created, its 
symbols are exported. This could make for a lot of duplicate symbols 
across multiple DLLs, though. Maybe there should be a method of 
explicitly exporting/importing a template instance from another DLL, e.g.

export alias symbol = templateFunc!int;

Please note, that the problem raised above by Benjamin applies just as 
well to non-exported templates.
Jan 27 2015
parent "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Tuesday, 27 January 2015 at 22:29:41 UTC, Rainer Schuetze 
wrote:

 I would not mind if we export all symbols on Windows aswell. It 
 doesn't seem to bother a lot of people for the linux version, 
 even though it's unsafer and slower than on Windows (at least 
 that was my experience more than 10 years ago). It might get us 
 a first working version without adding "export" throughout the 
 druntime/phobos source code.


There are multiple reasons why I don't want to simply export 
every symbol:

1) Before I started this implementation I synchronized with 
Martin Nowak regrading hish plans for D shared libraries. It 
turns out that he wants to annotate all of druntime and phobos 
with export asap and use it to control symbol visibility on 
linux. He wants to get away from the "export everything" on linux 
because it hurts performance and prevents some optimizations.
2) Every data symbol that is considered for exporting adds a 
slight performance overhead through a additional indirection, 
even for static builds. That is because the compiler can't know 
if the symbol is imported or not, as export means both import and 
export at the same time. So if the compiler simply assumes that 
all symbols are exported this would add a lot of unnecessary 
overhead even in static builds. (but also in dynamic ones)
3) If we start with export everything on Windows now, it will be 
hared to go back to export only whats annotated.


 I don't have a clear favorite, but the second version makes it 
 clearer that visibility and protection are separate issues.

 A note on:

 export public void templateFunc(T)()


 I don't think it is well defined what exporting a template is 
 supposed to mean. My guess: whenever an instance of the 
 template is created, its symbols are exported. This could make 
 for a lot of duplicate symbols across multiple DLLs, though.


Obviously its not yet well defined. But we can define it. And you 
are right, it means that all instances are exported. And we need 
that behavior because otherwise you have to spray in -allinst 
everywhere. Believe me I tried.


 Maybe there should be a method of explicitly 
 exporting/importing a template instance from another DLL, e.g.

 export alias symbol = templateFunc!int;


I would rather not do that. You don't have to explicitly import 
template instances from static libraries either. We should try to 
keep the behavior of static and dynamic libraries as similar as 
possible. The ideal situation would be that you can simply 
compile something that was a static library into a dynamic one 
without doing any code changes (other then writing export: at the 
beginning of every file)
Jan 28 2015
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Tuesday, 20 January 2015 at 12:23:32 UTC, Benjamin Thaut wrote:

 2) Make export an attribute. If export is no longer an 
 protection level but instead an attribute this issue can easily 
 be solved by doing.

 export public void templateFunc(T)()
 {
   someHelperFunc();
 }

 export private void someHelperFunc()
 {

 }


Clearly the better solution.
Export and protection have something in common but are not 
identical.
Jan 30 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 30 January 2015 at 19:10:06 UTC, Martin Nowak wrote:

 On Tuesday, 20 January 2015 at 12:23:32 UTC, Benjamin Thaut 
 wrote:

 2) Make export an attribute. If export is no longer an 
 protection level but instead an attribute this issue can 
 easily be solved by doing.

 export public void templateFunc(T)()
 {
  someHelperFunc();
 }

 export private void someHelperFunc()
 {

 }


 Clearly the better solution.
 Export and protection have something in common but are not 
 identical.


It has a serious drawback of increasing attribute noise even more 
though. First approach allows for more automatic inference.

But with D restrictions it seems the least bad option.
Jan 30 2015
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 31.01.2015 um 06:11 schrieb Dicebot:

 On Friday, 30 January 2015 at 19:10:06 UTC, Martin Nowak wrote:

 It has a serious drawback of increasing attribute noise even more
 though. First approach allows for more automatic inference.

 But with D restrictions it seems the least bad option.


Well, export is going to remain transitive. So the first approach is 
still going to work. The only difference is going to be that you can 
"force export" private declarations. So for most modules it is hopefully 
going to be enough to put "export { }"  around the public part of the 
module and force export some of the needed private declarations. For a 
module without templates a single "export { }" should be enough.
Jan 31 2015
parent reply "Martin Nowak" <code dawg.eu> writes:
On Saturday, 31 January 2015 at 09:25:10 UTC, Benjamin Thaut 
wrote:

 Well, export is going to remain transitive. So the first 
 approach is still going to work. The only difference is going 
 to be that you can "force export" private declarations. So for 
 most modules it is hopefully going to be enough to put "export 
 { }"  around the public part of the module and force export 
 some of the needed private declarations. For a module without 
 templates a single "export { }" should be enough.


That's probably how it should behave, though an attribute 
applying only to public members unless explicitly added is 
unprecedented. Still seems like the right choice here, but might 
require some additional compiler logic.
Jan 31 2015
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 31.01.2015 um 13:07 schrieb Martin Nowak:

 That's probably how it should behave, though an attribute applying only
 to public members unless explicitly added is unprecedented. Still seems
 like the right choice here, but might require some additional compiler
 logic.


Well you don't have to implement it that way with in the compiler. The 
only thing that matters is, that the users sees this transitive behavior 
of export. It is not neccessary that the attribute actually gets applied 
recursivly within the compiler implementation. Only the export behavior 
needs to be implemented recursivly and I already did a implementation 
for that. It currently works for the export protection level and would 
be trivial to adapt for a export attribute. It would also not require 
any additional logic for applying attributes recursivly under certain 
conditions.

 Walter:
Making export a attribute seems to be the preferred choice in this 
discussion. Additionaly this was the result of the last discussion 
around export, a year ago, although for different reasons. The last 
Discussion resulted in DIP 45 which also proposes making export an 
attribute. Before I start with the implementation, would you be ok with 
making export an attribute or would you veto it?

Kind Regards
Benjamin Thaut
Jan 31 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/31/2015 11:52 AM, Benjamin Thaut wrote:

  Walter:
 Making export a attribute seems to be the preferred choice in this discussion.
 Additionaly this was the result of the last discussion around export, a year
 ago, although for different reasons. The last Discussion resulted in DIP 45
 which also proposes making export an attribute. Before I start with the
 implementation, would you be ok with making export an attribute or would you
 veto it?


At this point I suggest simply making those private helper functions public and 
export them. It gets your project moving without waiting for language changes 
(and this is a breaking change).
Feb 16 2015
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 16 February 2015 at 08:08:17 UTC, Walter Bright wrote:

 On 1/31/2015 11:52 AM, Benjamin Thaut wrote:

  Walter:
 Making export a attribute seems to be the preferred choice in 
 this discussion.
 Additionaly this was the result of the last discussion around 
 export, a year
 ago, although for different reasons. The last Discussion 
 resulted in DIP 45
 which also proposes making export an attribute. Before I start 
 with the
 implementation, would you be ok with making export an 
 attribute or would you
 veto it?


 At this point I suggest simply making those private helper 
 functions public and export them. It gets your project moving 
 without waiting for language changes (and this is a breaking 
 change).


This is not about some private project but about fixing language 
itself.
Feb 16 2015
prev sibling next sibling parent reply "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Monday, 16 February 2015 at 08:08:17 UTC, Walter Bright wrote:

 At this point I suggest simply making those private helper 
 functions public and export them. It gets your project moving 
 without waiting for language changes (and this is a breaking 
 change).


This is in fact not a breaking change because export is broken 
anyway. Due to bug 922 export can't be used in cross platform 
libraries. I haven't seen a single library on dub that uses 
export (most likely for exactly that reason).
Also export on windows is broken as well, see Bug 9816.

So making export an attribute would most likely not break 
anything because it already is broken and wouldn't work if you 
tried to use it.
Feb 16 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/16/2015 1:40 AM, Benjamin Thaut wrote:

 On Monday, 16 February 2015 at 08:08:17 UTC, Walter Bright wrote:

 At this point I suggest simply making those private helper functions public
 and export them. It gets your project moving without waiting for language
 changes (and this is a breaking change).


 This is in fact not a breaking change because export is broken anyway. Due to
 bug 922 export can't be used in cross platform libraries. I haven't seen a
 single library on dub that uses export (most likely for exactly that reason).
 Also export on windows is broken as well, see Bug 9816.

 So making export an attribute would most likely not break anything because it
 already is broken and wouldn't work if you tried to use it.


 From https://issues.dlang.org/show_bug.cgi?id=9816:

----------
Here is a list of all things wrong with export:

32 & 64 bit issues:
1) Exporting a global variable leads to a linker error
2) Exporting thread local variables should be an error (at least it is in c++)
3) The module info should be exported as soon the module has any exported
symbols
4) __gshared members of a class are not exported
5) The TypeInfo Object of the TestClass is not exported
6) The TypeInfo Object of TestStruct is not exported
---------

None of these are addressed by making export an attribute.
Feb 16 2015
parent "Benjamin Thaut" <code benjamin-thaut.de> writes:
On Monday, 16 February 2015 at 09:59:07 UTC, Walter Bright wrote:

 ----------
 Here is a list of all things wrong with export:

 32 & 64 bit issues:
 1) Exporting a global variable leads to a linker error
 2) Exporting thread local variables should be an error (at 
 least it is in c++)
 3) The module info should be exported as soon the module has 
 any exported
 symbols
 4) __gshared members of a class are not exported
 5) The TypeInfo Object of the TestClass is not exported
 6) The TypeInfo Object of TestStruct is not exported
 ---------

 None of these are addressed by making export an attribute.


I never said that. I said that making export an attribute is 
_not_ an breaking change because it is already broken up to a 
point where it currently can't be used anyway (especially because 
of 922). My current implementation fixes both 922 and 9816 but 
would greatly benefit from making export an attribute because 
otherwise D's entire protection system would be undermined like 
described multiple times throughout this thread.
Feb 16 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-02-16 10:40, Benjamin Thaut wrote:


 This is in fact not a breaking change because export is broken anyway.
 Due to bug 922 export can't be used in cross platform libraries. I
 haven't seen a single library on dub that uses export (most likely for
 exactly that reason).


Most likely due to that the support for dynamic libraries is fairly new 
only works on Linux (?).

-- 
/Jacob Carlborg
Feb 16 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/16/2015 12:19 PM, Jacob Carlborg wrote:

 Most likely due to that the support for dynamic libraries is fairly new only
 works on Linux (?).


Part of the test suite creates a DLL using D for Windows.
Feb 16 2015
parent "Joakim" <dlang joakim.fea.st> writes:
On Monday, 16 February 2015 at 21:41:00 UTC, Walter Bright wrote:

 On 2/16/2015 12:19 PM, Jacob Carlborg wrote:

 Most likely due to that the support for dynamic libraries is 
 fairly new only
 works on Linux (?).


 Part of the test suite creates a DLL using D for Windows.


Martin recently added shared library support for FreeBSD also:

https://github.com/D-Programming-Language/druntime/pull/1068
Feb 16 2015
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 16 February 2015 at 20:19:27 UTC, Jacob Carlborg wrote:

 On 2015-02-16 10:40, Benjamin Thaut wrote:


 This is in fact not a breaking change because export is broken 
 anyway.
 Due to bug 922 export can't be used in cross platform 
 libraries. I
 haven't seen a single library on dub that uses export (most 
 likely for
 exactly that reason).


 Most likely due to that the support for dynamic libraries is 
 fairly new only works on Linux (?).


And the way it currently works on Linux `export` is completely 
ignored at all - hard to break something that has never worked.
Feb 18 2015
prev sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
So i looked at the Dll Test within the dmd test framework and when I 
make export an attribute, like I suggested, this perticular test will 
compile & run without any code changes. This is another reason why I 
suspect that making export an attribute will only break very little if 
any code at all.

Am 16.02.2015 um 09:08 schrieb Walter Bright:

 At this point I suggest simply making those private helper functions
 public and export them. It gets your project moving without waiting for
 language changes (and this is a breaking change).


I have to big fears with doing this

1) I finish everything up any finally do the pull request. Then the 
reviewers will realize that using export in all required places will 
completely undermine D's module level protection system and reject the 
PR. Then all my work is in vain.

2) Even if 1 does not happen, that means from now on the broken export 
will be used in _actual_ code. Because it kind of works. Going forward 
this would mean that changing export into an attribute will break actual 
production code. So in my opinion we have to implement Dll Support on 
windows and fix export simulatiously, otherwise fixing export will lead 
to big breaking changes instead of a few small ones.

Kind Regards
Benjamin Thaut
Feb 17 2015
parent "Andre" <andre s-e-a-p.de> writes:
Hi,

I also want to say a big thank you to all of you involved
in this topic and especially to Benjamin.

Proper DLL handling in D I would really appreciate.
I think this topic is the break through.

Kind regards
André

On Tuesday, 17 February 2015 at 18:03:06 UTC, Benjamin Thaut 
wrote:

 So i looked at the Dll Test within the dmd test framework and 
 when I make export an attribute, like I suggested, this 
 perticular test will compile & run without any code changes. 
 This is another reason why I suspect that making export an 
 attribute will only break very little if any code at all.

 Am 16.02.2015 um 09:08 schrieb Walter Bright:

 At this point I suggest simply making those private helper 
 functions
 public and export them. It gets your project moving without 
 waiting for
 language changes (and this is a breaking change).


 I have to big fears with doing this

 1) I finish everything up any finally do the pull request. Then 
 the reviewers will realize that using export in all required 
 places will completely undermine D's module level protection 
 system and reject the PR. Then all my work is in vain.

 2) Even if 1 does not happen, that means from now on the broken 
 export will be used in _actual_ code. Because it kind of works. 
 Going forward this would mean that changing export into an 
 attribute will break actual production code. So in my opinion 
 we have to implement Dll Support on windows and fix export 
 simulatiously, otherwise fixing export will lead to big 
 breaking changes instead of a few small ones.

 Kind Regards
 Benjamin Thaut
Feb 18 2015