www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Invalid memory operation during allocation with `new`

reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
My application has suddendly started crashing as

core.exception.InvalidMemoryOperationError src/core/exception.d(647): Invalid
memory operation

during a plain allocation with `new` during execution of `main`.

The crash happens only if the `unittests` are run prior to `main`.

The crash goes away when I disable at least on the `unittest`s 
run before `main`.

I'm compiling with DMD version 2.090.

How do I check the amount of memory currently being used in a 
process?

If the amount of memory used is not the problem, how can I find 
the reason for this?
Jan 12 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 12 January 2020 at 13:39:42 UTC, Per Nordlöw wrote:

 core.exception.InvalidMemoryOperationError src/core/exception.d(647): Invalid
memory operation


That means a destructor tried to perform a GC operation while the 
GC was running.


 The crash happens only if the `unittests` are run prior to 
 `main`.


Check all the classes created by those unittests. If any of them 
have destructors that allocate memory in any way - including 
calling like `writeln(this)` cuz that can call toString, change 
that.
Jan 12 2020
parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Sunday, 12 January 2020 at 13:58:25 UTC, Adam D. Ruppe wrote:

 Check all the classes created by those unittests. If any of 
 them have destructors that allocate memory in any way - 
 including calling like `writeln(this)` cuz that can call 
 toString, change that.


Thanks. So what about,

- in the short run,
   Qualifying all user-defined class destructors with ` nogc`?

- in the long run,
   Making DMD forbid GC-allocations transitively inside class 
destructors?
Jan 12 2020
next sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Sunday, 12 January 2020 at 15:21:05 UTC, Per Nordlöw wrote:

 Thanks. So what about,

 - in the short run,
   Qualifying all user-defined class destructors with ` nogc`?

 - in the long run,
   Making DMD forbid GC-allocations transitively inside class 
 destructors?


AFAICT, these rules should in most cases hold also for struct 
destructors as those might be called inside class destructors, 
transitively.
Jan 12 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 12 January 2020 at 15:21:05 UTC, Per Nordlöw wrote:

 - in the short run,
   Qualifying all user-defined class destructors with ` nogc`?

 - in the long run,
   Making DMD forbid GC-allocations transitively inside class 
 destructors?


The class doesn't necessarily know what its destructor is going 
to be used for, nor does as struct. So such static checks would 
end problematic; limiting to some legit cases and not catching 
some problem cases (because destructors are called with child 
classes to).

The most recent release added a runtime function you can test 
though:

https://dlang.org/phobos/core_memory.html#.GC.inFinalizer

so if that is true, avoid doing the call.

Of course that doesn't help the case when you don't know the rule 
and don't even think to call it.......
Jan 12 2020
parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Sunday, 12 January 2020 at 17:36:23 UTC, Adam D. Ruppe wrote:

 Of course that doesn't help the case when you don't know the 
 rule and don't even think to call it.......


Thanks. I checked and  nogc-qualified all the class destructors 
defined in my project. Without finding any potential problems. 
Any other suggestions on how to trace this problem?
Jan 13 2020
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/13/20 4:34 AM, Per Nordlöw wrote:

 On Sunday, 12 January 2020 at 17:36:23 UTC, Adam D. Ruppe wrote:

 Of course that doesn't help the case when you don't know the rule and 
 don't even think to call it.......


 
 Thanks. I checked and  nogc-qualified all the class destructors defined 
 in my project. Without finding any potential problems. Any other 
 suggestions on how to trace this problem?


Finding where this is happening is a really difficult problem. Not only 
because there is no stack trace (I believe this is because the invalid 
memory operation error cannot allocate any GC memory, and there are very 
small parts of the stack trace printing code that allocate), but because 
it is invariably happening inside a GC collection -- so the error is far 
away from where you allocated (sometimes the conditions that trigger the 
dtor to allocate are hard to figure out from looking at the object when 
it's destroyed).

I actually had a problem in my vibe.d code where the error didn't 
display until the system was shutting down. Figuring out where it was 
involved defining the extern(C) function onInvalidMemoryOperationError 
(I think if you define this in your code, it overrides the runtime 
library's version), so I could do some diagnostics, and set a breakpoint.

Try starting there and see if you can find why the IMO is happening.

https://github.com/dlang/druntime/blob/c85dea29cb721ec76af6793076aa93e2f62a23da/src/core/exception.d#L535

-Steve
Jan 13 2020
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 13 January 2020 at 09:34:42 UTC, Per Nordlöw wrote:

 Without finding any potential problems. Any other suggestions 
 on how to trace this problem?


Run the program in a debugger with the `--DRT-trapExceptions=0` 
argument to the program with certain versions of druntime, or set 
a breakpoint at that function.

On Linux for example, I made a program "pain" that deliberately 
crashes:

-----------------

$ gdb pain
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-slackware-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
     <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from pain...
(gdb) break onInvalidMemoryOperationError
Breakpoint 1 at 0x4614f8
(gdb) r
Starting program: /home/me/test/pain
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, 0x00000000004614f8 in onInvalidMemoryOperationError 
()
(gdb) where


_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs12mallocNoSyncMFNbmkKmxC8TypeInfoZPvS_DQEgQEgQEeQEp10mallocTimelS_DQFiQFiQFgQFr10numMallocslTmTkTmTxQCzZQF
MFNbKmKkKmKxQDsZQDl ()

_D2gc4impl12conservativeQw14ConservativeGC6mallocMFNbmkxC8TypeInfoZPv ()



at pain.d:3



_D2gc4impl12conservativeQw3Gcx5sweepMFNbZm ()

_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm ()

_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs18fullCollectNoStackMFNbZ2goFNbPSQEaQEaQDyQEj3Gc
ZmTQvZQDfMFNbKQBgZm ()

_D2gc4impl12conservativeQw14ConservativeGC18fullCollectNoStackMFNbZv ()

_D2gc4impl12conservativeQw14ConservativeGC14collectNoStackMFNbZv 
()



_D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv ()

_D2rt6dmain212_d_run_main2UAAamPUQgZiZ7tryExecMFMDFZvZv ()




/lib64/libc.so.6
(gdb)

-----------------


So, first make a debug build of the program (dmd -g pain.d). 
Then, run the program in the debugger:

gdb ./pain

And then ask the debugger to break on this invalid operation:

break onInvalidMemoryOperationError

NOTE: the onInvalidMemoryOperationError is actually a function 
time inside druntime.

Then run the program with gdb's `r` command.

It breaks when that function is called, right before it actually 
would throw

Breakpoint 1, 0x00000000004614f8 in onInvalidMemoryOperationError 
()

So then I asked the debugger where this occurred:

(gdb) where

And visually scanned for a function I recognized (and my gdb also 
conveniently highlighted the filename in green)


at pain.d:3

That looks like my code! pain.d line 3 caused the problem.

class Foo {
         ~this() {
                 auto a = new Object(); // this
         }
}

void main() {
         auto foo = new Foo();
}


It might not be this simple in a real bug situation - this 
program was specifically written to cause it - but the same 
approach should narrow it down pretty quickly for you.
Jan 13 2020
parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Monday, 13 January 2020 at 20:30:33 UTC, Adam D. Ruppe wrote:

 Reading symbols from pain...
 (gdb) break onInvalidMemoryOperationError
 Breakpoint 1 at 0x4614f8
 (gdb) r
 Starting program: /home/me/test/pain
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library "/lib64/libthread_db.so.1".


Thanks a lot.

Unfortunately that didn't tell me the cause of the problem but 
only the stack trace of the failing call to `new` which is not 
called inside a destructor.
Jan 13 2020
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/13/20 6:06 PM, Per Nordlöw wrote:

 On Monday, 13 January 2020 at 20:30:33 UTC, Adam D. Ruppe wrote:

 Reading symbols from pain...
 (gdb) break onInvalidMemoryOperationError
 Breakpoint 1 at 0x4614f8
 (gdb) r
 Starting program: /home/me/test/pain
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library "/lib64/libthread_db.so.1".


 
 Thanks a lot.
 
 Unfortunately that didn't tell me the cause of the problem but only the 
 stack trace of the failing call to `new` which is not called inside a 
 destructor.


can you post the stack trace?

-Steve
Jan 14 2020
prev sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Monday, 13 January 2020 at 23:06:01 UTC, Per Nordlöw wrote:

 Unfortunately that didn't tell me the cause of the problem but 
 only the stack trace of the failing call to `new` which is not 
 called inside a destructor.


I found the problem. I was greedly using the `__monitor__` slot 
of some classes in order to get one extra word of class storage. 
Setting it to `null` in those classes' destructors made the 
memory error go away. Thanks anyway for valuable input.
Jan 15 2020