www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - how to call std_stdio_static_this from a dynamically loaded shared

reply "timotheecour" <thelastmammoth gmail.com> writes:
How do I call std_stdio_static_this() in std.stdio from a 
dynamically loaded shared library (osx) ?

I need to do this in my scenario:
1) main program is launched
2) dynamic library is created
3) dynamic library is loaded and a function from it is called
4) everything works fine until a writeln(0) is called from a 
function inside the dynamic lib, at which point it crashes. But 
by calling std_stdio_static_this2 (see below) BEFORE any call to 
writeln (and similar functions), everything works fine.

When I call std_stdio_static_this from another module, I'm pretty 
sure that code doesn't get called, because when I add a printf or 
assert(0) statement inside std_stdio_static_this, nothing happens.

My current (working!) workaround is shown below, but I was hoping 
for a solution that didn't involve modifying std.stdio.

----
void std_stdio_static_this2(T=void)()
{
     //copy contents of std_stdio_static_this() function here:
     //putting assert(0); here WILL crash, as expected; but it 
won't if the signature is std_stdio_static_this2().
     //Bind stdin, stdout, stderr
     __gshared File.Impl stdinImpl;
     stdinImpl.handle = core.stdc.stdio.stdin;
     .stdin.p = &stdinImpl;
     // stdout
     __gshared File.Impl stdoutImpl;
     stdoutImpl.handle = core.stdc.stdio.stdout;
     .stdout.p = &stdoutImpl;
     // stderr
     __gshared File.Impl stderrImpl;
     stderrImpl.handle = core.stdc.stdio.stderr;
     .stderr.p = &stderrImpl;
}
----
Oct 02 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-03 02:55, timotheecour wrote:
 How do I call std_stdio_static_this() in std.stdio from a dynamically
 loaded shared library (osx) ?

 I need to do this in my scenario:
 1) main program is launched
 2) dynamic library is created
 3) dynamic library is loaded and a function from it is called
 4) everything works fine until a writeln(0) is called from a function
 inside the dynamic lib, at which point it crashes. But by calling
 std_stdio_static_this2 (see below) BEFORE any call to writeln (and
 similar functions), everything works fine.

Dynamic libraries are not properly working, see this thread: http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com And this post: http://forum.dlang.org/thread/k3vfm9$1tq$1 digitalmars.com?page=3#post-k4219f:24uft:241:40digitalmars.com -- /Jacob Carlborg
Oct 02 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-03 08:56, timotheecour wrote:
 Thanks for the links. Ya, I did see those threads and I understand there
 are limitations with dynamic load shared libraries, however, your answer
 is not very helpful. We shouldn't have to wait until they get 100%
 support to start using them (who knows when that happens); some
 applications just require them (eg writing matlab mex files).

Yeah, it may not have been so helpful, sorry.
 In the meantime, the workaround I described above seems to adequately
 address the IO problem in shared libraries, it's just that it requires
 hacking inside std.stdio. So I'm just wondering whether I should:
 * pull those changes back in phobos
 * or if there's an alternative way of doing this

Module constructors if one of the things that don't work properly. I don't know exactly how this works in std.stdio I would need to study the code. If I recall correctly std.stdio uses some kind of hack to avoid circular dependencies that can otherwise occur if two modules, both having module constructors, import each other. -- /Jacob Carlborg
Oct 03 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-03 20:23, timotheecour wrote:

 Yes, that hack consists in :
 1) defining extern(C) void std_stdio_static_this() instead of static this()
 2) defining an auxiliary module std.stdiobase who will call it at module
 construction:
 module std.stdiobase;
 extern(C) void std_stdio_static_this();
 shared static this(){
      std_stdio_static_this();
 }

 I'm still puzzled as to why modifying the code inside
 std_stdio_static_this (say with assert(0)) doesn't have any effect
 (hence the need for the templated version I'm using).
 Another workaround (besides adding std_stdio_static_this2) would be to
 make the field File.p non-private so that my std_stdio_static_this2
 could be defined outside std.stdio. Any suggestion?

What happens if you just call "std_stdio_static_this" the first you do in your main function? Usually these kinds of problems are solved by using lazy initialization. In this case that would mean converting "stdin", "stdout" and "stderr" to functions. https://github.com/D-Programming-Language/phobos/blob/master/std/stdio.d#L2357 Something like this: private __gshared { File _stdin; File _stdout; File _stderr; File.Impl stdinImpl; File.Impl stdoutImpl; File.Impl stderrImpl; } property File stdin () { if (!_stdin.p) _stdin.p = &stdinImpl; return _stdin; } The same for "stdout" and "stderr". But this may not be good for performance reasons. I don't know how good the compiler is at optimizing these kind of functions. -- /Jacob Carlborg
Oct 03 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-03 23:51, timotheecour wrote:

 In my case I can't: I don't have control over the main function (it
 could be written in C for example).

If you're developing a library or similar you could have a requirement that the user need to call "Library.initialize()" before use. But of course, that's something anyone would like to avoid.
 I guess you also want to add inside the if(): "stdinImpl.handle =
 core.stdc.stdio.stdin;", etc.

Right, I missed that.
 I doubt the compiler can do much optimizing here, since the if won't
 always be true, but probably IO is slow enough compared to a branch so
 it might not matter.

 That being said, templatizing std_stdio_static_this as above would avoid
 branching at every single call to writeXX functions. Would there be any
 disadvantage?

I have no idea. I would guess as long as "std_stdio_static_this" or the corresponding code is run before std.stdio is used you should be fine. You have to be careful here about threads, you don't want to have a thread you don't know about sneaking up behind your back and starts using std.stdio before it's initialized.
 Also, I grepped for static_this, it appears this hack is only used in 2
 places:
 * std.stdio (=>stdiobase)
 * std.process (=>processinit)
 So the same template trick could be used in those 2 places.

You can write this instead: void std_stdio_static_this2 () () Note the extra pair of empty parentheses. This will force the function to be a template. I think this look slightly better. -- /Jacob Carlborg
Oct 03 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-04 20:13, timotheecour wrote:

 I'm fine with that (and already doing this, to call Runtime.initialize),
 but that doesn't solve the issue of this thread.

No, not if you need to redefine functions. -- /Jacob Carlborg
Oct 04 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-10-04 21:30, timotheecour wrote:

 Even if I don't need to redefine:

 As I was explaining in the first post, my original problem was that
 calling the existing (non-templated, non-modified) extern(C)
 std_stdio_static_this() seems to have no effect: a call to writeln(0)
 will cause a runtime crash in the shared lib.

Yes, that's what how I understood it. Do you want to do both?
 Even after a call to Runtime.initialize. So far, only adding a templated
 version std_stdio_static_this2()() and calling it allows writeln(0) to
 work.

 I wish somebody had an explanation why std_stdio_static_this() doesn't
 work but std_stdio_static_this2()() works.

I have no idea. Do you have a small test case to show, including how you compiled it? -- /Jacob Carlborg
Oct 04 2012
prev sibling next sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
Thanks for the links. Ya, I did see those threads and I 
understand there are limitations with dynamic load shared 
libraries, however, your answer is not very helpful. We shouldn't 
have to wait until they get 100% support to start using them (who 
knows when that happens); some applications just require them (eg 
writing matlab mex files).

In the meantime, the workaround I described above seems to 
adequately address the IO problem in shared libraries, it's just 
that it requires hacking inside std.stdio. So I'm just wondering 
whether I should:
* pull those changes back in phobos
* or if there's an alternative way of doing this

Thanks!
Oct 02 2012
prev sibling next sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
 Module constructors if one of the things that don't work 
 properly. I don't know exactly how this works in std.stdio I 
 would need to study the code. If I recall correctly std.stdio 
 uses some kind of hack to avoid circular dependencies that can 
 otherwise occur if two modules, both having module 
 constructors, import each other.

Yes, that hack consists in : 1) defining extern(C) void std_stdio_static_this() instead of static this() 2) defining an auxiliary module std.stdiobase who will call it at module construction: module std.stdiobase; extern(C) void std_stdio_static_this(); shared static this(){ std_stdio_static_this(); } I'm still puzzled as to why modifying the code inside std_stdio_static_this (say with assert(0)) doesn't have any effect (hence the need for the templated version I'm using). Another workaround (besides adding std_stdio_static_this2) would be to make the field File.p non-private so that my std_stdio_static_this2 could be defined outside std.stdio. Any suggestion?
Oct 03 2012
prev sibling next sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
 What happens if you just call "std_stdio_static_this" the first 
 you do in your main function?

In my case I can't: I don't have control over the main function (it could be written in C for example).
  property File stdin ()
 {
     if (!_stdin.p)
         _stdin.p = &stdinImpl;
     return _stdin;
 }

I guess you also want to add inside the if(): "stdinImpl.handle = core.stdc.stdio.stdin;", etc.
 The same for "stdout" and "stderr". But this may not be good 
 for performance reasons. I don't know how good the compiler is 
 at optimizing these kind of functions.

I doubt the compiler can do much optimizing here, since the if won't always be true, but probably IO is slow enough compared to a branch so it might not matter. That being said, templatizing std_stdio_static_this as above would avoid branching at every single call to writeXX functions. Would there be any disadvantage? Also, I grepped for static_this, it appears this hack is only used in 2 places: * std.stdio (=>stdiobase) * std.process (=>processinit) So the same template trick could be used in those 2 places.
Oct 03 2012
prev sibling next sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
I moved the discussion to 
http://forum.dlang.org/thread/iegurltmszdndxqnoyxw forum.dlang.org#post-iegurltmszdndxqnoyxw:40forum.dlang.org
since the problem appears more general, ie how to redefine an 
extern(C) function in a dynamic load library
Oct 03 2012
prev sibling next sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
 If you're developing a library or similar you could have a 
 requirement that the user need to call "Library.initialize()" 
 before use. But of course, that's something anyone would like 
 to avoid.

I'm fine with that (and already doing this, to call Runtime.initialize), but that doesn't solve the issue of this thread.
 You can write this instead:
 void std_stdio_static_this2 () ()

Thanks! I didn't know that.
Oct 04 2012
prev sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
On Thursday, 4 October 2012 at 19:20:54 UTC, Jacob Carlborg wrote:
 On 2012-10-04 20:13, timotheecour wrote:

 I'm fine with that (and already doing this, to call 
 Runtime.initialize),
 but that doesn't solve the issue of this thread.


 No, not if you need to redefine functions.

Even if I don't need to redefine: As I was explaining in the first post, my original problem was that calling the existing (non-templated, non-modified) extern(C) std_stdio_static_this() seems to have no effect: a call to writeln(0) will cause a runtime crash in the shared lib. Even after a call to Runtime.initialize. So far, only adding a templated version std_stdio_static_this2()() and calling it allows writeln(0) to work. I wish somebody had an explanation why std_stdio_static_this() doesn't work but std_stdio_static_this2()() works.
Oct 04 2012