www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - OutOfMemoryError in D DLL appending to module-level array

reply cc <cc nevernet.com> writes:
Ordinarily, it seems legal to append to an array that has been 
declared at module level (or as a static class member) that 
hasn't been otherwise initialized, for example:
```d
class Foo {}
private Foo[] cache;
void main() {
	auto foo = new Foo();
	cache ~= foo;
}
```

However, when building code like this as a DLL, such as:
```d
class Foo {}
private Foo[] cache;
extern(C) export Foo createFoo() {
	auto foo = new Foo();
	cache ~= foo;
	return foo;
}
```

I get `core.exception.OutOfMemoryError src\core\exception.d(647): 
Memory allocation failed` at the `cache ~= foo;` line.

I was able to get around this by adding:
```d
static this() {
	cache.length = 0;
}
```
which seems to fix it, but I'm not entirely sure what's going on, 
if this is expected behavior, if that's the correct way to handle 
it, and so on.  Does it have something to do with the D runtime 
being initialized differently in a DLL versus a statically linked 
program?  I am calling Runtime.initialize() as expected when the 
DLL is attached.
May 01 2021
next sibling parent cc <cc nevernet.com> writes:
On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:

 [...]


Just to add, only appending to the array seems to give 
OutOfMemoryErrors.  I can idup strings, call stdc malloc, etc 
just fine.
May 01 2021
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:

 which seems to fix it, but I'm not entirely sure what's going 
 on, if this is expected behavior, if that's the correct way to 
 handle it, and so on.


Oh I've been working on this the last couple weeks and having a 
hard time reproducing outside the work application. In the work 
app, the GC wasn't scanning the dll's TLS variables and freeing 
them prematurely.

In a sample test program, I used a thing kinda like yours, if a 
dll creates a thread and calls back into the exe you get a 
separate problem of partially initialize data.


D dlls on Windows work in simple cases right now but break down 
in more advanced cases. The good news is there's major fixes 
coming soon - my druntime hack might be coming, gdc is getting 
full dll support very soon from mingw, there's a good chance ldc 
is going to in a release or two as well outside mingw.

But the bad news is none of that is actually out right now, so 
dll + tls variables (which includes the top-level things on 
modules) are potentially buggy among other things like duplicated 
symbols.

You might find some improvement making your variable __gshared 
there.

But if you can do any reduced test case I'd really appreciate it. 
More tests that we can do in public is better!
May 01 2021
parent cc <cc nevernet.com> writes:
On Sunday, 2 May 2021 at 02:42:46 UTC, Adam D. Ruppe wrote:

 On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:

 which seems to fix it, but I'm not entirely sure what's going 
 on, if this is expected behavior, if that's the correct way to 
 handle it, and so on.


 Oh I've been working on this the last couple weeks and having a 
 hard time reproducing outside the work application. In the work 
 app, the GC wasn't scanning the dll's TLS variables and freeing 
 them prematurely.

 In a sample test program, I used a thing kinda like yours, if a 
 dll creates a thread and calls back into the exe you get a 
 separate problem of partially initialize data.


 D dlls on Windows work in simple cases right now but break down 
 in more advanced cases. The good news is there's major fixes 
 coming soon - my druntime hack might be coming, gdc is getting 
 full dll support very soon from mingw, there's a good chance 
 ldc is going to in a release or two as well outside mingw.

 But the bad news is none of that is actually out right now, so 
 dll + tls variables (which includes the top-level things on 
 modules) are potentially buggy among other things like 
 duplicated symbols.

 You might find some improvement making your variable __gshared 
 there.

 But if you can do any reduced test case I'd really appreciate 
 it. More tests that we can do in public is better!


Cool, thanks for the update.  Setting it as __gshared does seem 
to work.  I put together some test cases here:

https://gitlab.com/-/snippets/2114152

It's got the DLL written in D, and test programs for loading it 

seems to work.  I'd welcome any recommendations on how to improve 
the interfaces if there are any, I made a little mixin to create 
C wrappers for the member functions since that seems to be the 
suggested solution for calling class methods.
May 01 2021