www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - OutOfMemoryError in D DLL appending to module-level array

reply cc <cc nevernet.com> writes:
Ordinarily, it seems legal to append to an array that has been 
declared at module level (or as a static class member) that 
hasn't been otherwise initialized, for example:
```d
class Foo {}
private Foo[] cache;
void main() {
	auto foo = new Foo();
	cache ~= foo;
}
```

However, when building code like this as a DLL, such as:
```d
class Foo {}
private Foo[] cache;
extern(C) export Foo createFoo() {
	auto foo = new Foo();
	cache ~= foo;
	return foo;
}
```

I get `core.exception.OutOfMemoryError src\core\exception.d(647): 
Memory allocation failed` at the `cache ~= foo;` line.

I was able to get around this by adding:
```d
static this() {
	cache.length = 0;
}
```
which seems to fix it, but I'm not entirely sure what's going on, 
if this is expected behavior, if that's the correct way to handle 
it, and so on.  Does it have something to do with the D runtime 
being initialized differently in a DLL versus a statically linked 
program?  I am calling Runtime.initialize() as expected when the 
DLL is attached.
May 01
next sibling parent cc <cc nevernet.com> writes:
On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:
 [...]
Just to add, only appending to the array seems to give OutOfMemoryErrors. I can idup strings, call stdc malloc, etc just fine.
May 01
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:
 which seems to fix it, but I'm not entirely sure what's going 
 on, if this is expected behavior, if that's the correct way to 
 handle it, and so on.
Oh I've been working on this the last couple weeks and having a hard time reproducing outside the work application. In the work app, the GC wasn't scanning the dll's TLS variables and freeing them prematurely. In a sample test program, I used a thing kinda like yours, if a dll creates a thread and calls back into the exe you get a separate problem of partially initialize data. D dlls on Windows work in simple cases right now but break down in more advanced cases. The good news is there's major fixes coming soon - my druntime hack might be coming, gdc is getting full dll support very soon from mingw, there's a good chance ldc is going to in a release or two as well outside mingw. But the bad news is none of that is actually out right now, so dll + tls variables (which includes the top-level things on modules) are potentially buggy among other things like duplicated symbols. You might find some improvement making your variable __gshared there. But if you can do any reduced test case I'd really appreciate it. More tests that we can do in public is better!
May 01
parent cc <cc nevernet.com> writes:
On Sunday, 2 May 2021 at 02:42:46 UTC, Adam D. Ruppe wrote:
 On Sunday, 2 May 2021 at 02:34:41 UTC, cc wrote:
 which seems to fix it, but I'm not entirely sure what's going 
 on, if this is expected behavior, if that's the correct way to 
 handle it, and so on.
Oh I've been working on this the last couple weeks and having a hard time reproducing outside the work application. In the work app, the GC wasn't scanning the dll's TLS variables and freeing them prematurely. In a sample test program, I used a thing kinda like yours, if a dll creates a thread and calls back into the exe you get a separate problem of partially initialize data. D dlls on Windows work in simple cases right now but break down in more advanced cases. The good news is there's major fixes coming soon - my druntime hack might be coming, gdc is getting full dll support very soon from mingw, there's a good chance ldc is going to in a release or two as well outside mingw. But the bad news is none of that is actually out right now, so dll + tls variables (which includes the top-level things on modules) are potentially buggy among other things like duplicated symbols. You might find some improvement making your variable __gshared there. But if you can do any reduced test case I'd really appreciate it. More tests that we can do in public is better!
Cool, thanks for the update. Setting it as __gshared does seem to work. I put together some test cases here: https://gitlab.com/-/snippets/2114152 It's got the DLL written in D, and test programs for loading it seems to work. I'd welcome any recommendations on how to improve the interfaces if there are any, I made a little mixin to create C wrappers for the member functions since that seems to be the suggested solution for calling class methods.
May 01