www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - What is the Correct way to Malloc in nogc section?

reply "Kitt" <NekoAssassin gmail.com> writes:
I'm currently trying to write a personal Associate Array class 
that can be used in  nogc sections. Obviously, this means I'll 
want to use malloc/free, however I'm not sure what the "Correct" 
way to do this is. In a few places online I've seen code for 
custom "_new" and "_delete" functions similar to this:

[code]
T _new(T, Args...) (Args args) {
     size_t objSize;
	static if(is (ValType == class))
	    objSize = __traits(classInstanceSize, T);
	else
	    objSize = T.sizeof;

     void* tmp = core.stdc.stdlib.malloc(objSize);
     if (!tmp) throw new Exception("Memory allocation failed");
     void[] mem = tmp[0..objSize];
     T obj = emplace!(T, Args)(mem, args);
     return obj;
}

void _delete(T)(T obj) {
     clear(obj);
     core.stdc.stdlib.free(cast(void*)obj);
}
[/code]

The Exception obviously uses the GC, and would need to be 
replaced with a printf or something; however, emplace doesn't 
have a  nogc replacement that I know of. What I'd like to know, 
from people much more knowledgeable about the ins and outs of D, 
is what the "Best" or "Correct" way to allocate and deallocate 
within  nogc sections?

PS: Tangentially related, what hashing algorithm does D use? I've 
seen people suggest using typeid(object).toHash(&object); 
however, toHash isn't  nogc, so I need to write my own Hashing 
function. For the sake of consistency and peace of mind, I'd 
really like to match mine as closely to Ds internal one as 
possible.
Feb 12 2015
next sibling parent "Kitt" <NekoAssassin gmail.com> writes:
I realized the above "_new" has an issue with emplace depending 
on whether it's a class (reference type) or not. Class c = 
emplace() works, but something like string s = emplace() does 
not. Is doing string s = *emplace() safe and okay? Or is there a 
better way to handle the differences between allocating a Class 
with malloc vs a non-Class?
Feb 12 2015
prev sibling next sibling parent reply "Foo" <Foo test.de> writes:
On Thursday, 12 February 2015 at 23:27:51 UTC, Kitt wrote:

 I'm currently trying to write a personal Associate Array class 
 that can be used in  nogc sections. Obviously, this means I'll 
 want to use malloc/free, however I'm not sure what the 
 "Correct" way to do this is. In a few places online I've seen 
 code for custom "_new" and "_delete" functions similar to this:

 [code]
 T _new(T, Args...) (Args args) {
     size_t objSize;
 	static if(is (ValType == class))
 	    objSize = __traits(classInstanceSize, T);
 	else
 	    objSize = T.sizeof;

     void* tmp = core.stdc.stdlib.malloc(objSize);
     if (!tmp) throw new Exception("Memory allocation failed");
     void[] mem = tmp[0..objSize];
     T obj = emplace!(T, Args)(mem, args);
     return obj;
 }

 void _delete(T)(T obj) {
     clear(obj);
     core.stdc.stdlib.free(cast(void*)obj);
 }
 [/code]

 The Exception obviously uses the GC, and would need to be 
 replaced with a printf or something; however, emplace doesn't 
 have a  nogc replacement that I know of. What I'd like to know, 
 from people much more knowledgeable about the ins and outs of 
 D, is what the "Best" or "Correct" way to allocate and 
 deallocate within  nogc sections?

 PS: Tangentially related, what hashing algorithm does D use? 
 I've seen people suggest using typeid(object).toHash(&object); 
 however, toHash isn't  nogc, so I need to write my own Hashing 
 function. For the sake of consistency and peace of mind, I'd 
 really like to match mine as closely to Ds internal one as 
 possible.


This is something I've done recently.
Would be glad if my code will help you:
https://github.com/Dgame/m3
Especially the m3.d module could be useful for you.
Feb 12 2015
parent reply "anonymous" <anonymous example.com> writes:
On Thursday, 12 February 2015 at 23:52:41 UTC, Foo wrote:

 This is something I've done recently.
 Would be glad if my code will help you:
 https://github.com/Dgame/m3
 Especially the m3.d module could be useful for you.


/* Class and Struct */

emplace: You can't assume zero-initialization for structs.
destruct: Is not memory-safe, and must not be marked  trusted.

/* Array */

make: You can't assume zero-initialization. T.sizeof is not the 
size of an element.
reserve: Not safe (you're freeing the old memory), must not be 
 trusted.
append: T.sizeof is not the size of an element. You're 
multiplying twice with T.sizeof; in `append`, and also in 
`reserve`.
destruct: Not safe, must not be  trusted.
Feb 13 2015
next sibling parent reply "Foo" <Foo test.de> writes:
On Friday, 13 February 2015 at 22:55:27 UTC, anonymous wrote:

 On Thursday, 12 February 2015 at 23:52:41 UTC, Foo wrote:

 This is something I've done recently.
 Would be glad if my code will help you:
 https://github.com/Dgame/m3
 Especially the m3.d module could be useful for you.


 /* Class and Struct */

 emplace: You can't assume zero-initialization for structs.
 destruct: Is not memory-safe, and must not be marked  trusted.

 /* Array */

 make: You can't assume zero-initialization. T.sizeof is not the 
 size of an element.


I'm aware of that.

 reserve: Not safe (you're freeing the old memory), must not be 
  trusted.
 append: T.sizeof is not the size of an element. You're 
 multiplying twice with T.sizeof; in `append`, and also in 
 `reserve`.
 destruct: Not safe, must not be  trusted.


Ah, that is nice to know. I will fix this soon. Or would you help 
me out with an PR?

Don't understand me wrong. My code is not perfect, but maybe it 
can be helpful for someone.
Feb 13 2015
parent reply "anonymous" <anonymous example.com> writes:
On Friday, 13 February 2015 at 23:04:25 UTC, Foo wrote:

 Don't understand me wrong. My code is not perfect, but maybe it 
 can be helpful for someone.


As it is right now, I fear it may do more harm than good.
Feb 13 2015
parent reply "Foo" <Foo test.de> writes:
On Friday, 13 February 2015 at 23:13:05 UTC, anonymous wrote:

 On Friday, 13 February 2015 at 23:04:25 UTC, Foo wrote:

 Don't understand me wrong. My code is not perfect, but maybe 
 it can be helpful for someone.


 As it is right now, I fear it may do more harm than good.


Always the same in this newsgroup. You want to help as good as 
you can (even if it's not perfect) and all you get are subliminal 
messages... :)

I'm regret that I tried to help, I will delete this repo as far 
as possible. :)
Feb 13 2015
next sibling parent "Kitt" <NekoAssassin gmail.com> writes:
On Friday, 13 February 2015 at 23:29:11 UTC, Foo wrote:

 On Friday, 13 February 2015 at 23:13:05 UTC, anonymous wrote:

 On Friday, 13 February 2015 at 23:04:25 UTC, Foo wrote:

 Don't understand me wrong. My code is not perfect, but maybe 
 it can be helpful for someone.


 As it is right now, I fear it may do more harm than good.


 Always the same in this newsgroup. You want to help as good as 
 you can (even if it's not perfect) and all you get are 
 subliminal messages... :)

 I'm regret that I tried to help, I will delete this repo as far 
 as possible. :)


Please don't do that. Even if the code has some errors, it was a 
helpful and useful module I can work with. I'm sure others may 
feel the same way =)

Stick and Stones.
Feb 13 2015
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Foo:


 I'm regret that I tried to help, I will delete this repo as far 
 as possible. :)


Language communities aren't perfect, but the success of a 
language comes from the help of many little hands :) Perhaps Rust 
will "win" over D in the end, but there's no reason to throw away 
your work just for a comment... be tolerant and be good.

Bye,
bearophile
Feb 13 2015
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Fri, 13 Feb 2015 23:29:10 +0000, Foo wrote:


 Always the same in this newsgroup. You want to help as good as you can
 (even if it's not perfect) and all you get are subliminal messages... :)


code that you throwing at newcomer may not be perfect, but it at least=20
should not be wrong. your code is obvious wrong, as anonymous said.

you may make your own question like "what is wrong with this code?".=20
would you be glad to get the wrong answer?

anonymous is right: your code does more harm than good *as* *an*=20
*answer*. but it seems that you don't care about the quality, "it works=20
for me so it's good", yeah? 'cause if you *do* care, you're able to see=20
why anonymous said what he's said.=
Feb 13 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Friday, 13 February 2015 at 22:55:27 UTC, anonymous wrote:

 On Thursday, 12 February 2015 at 23:52:41 UTC, Foo wrote:

 This is something I've done recently.
 Would be glad if my code will help you:
 https://github.com/Dgame/m3
 Especially the m3.d module could be useful for you.


 /* Class and Struct */

 emplace: You can't assume zero-initialization for structs.
 destruct: Is not memory-safe, and must not be marked  trusted.

 /* Array */

 make: You can't assume zero-initialization. T.sizeof is not the 
 size of an element.
 reserve: Not safe (you're freeing the old memory), must not be 
  trusted.
 append: T.sizeof is not the size of an element. You're 
 multiplying twice with T.sizeof; in `append`, and also in 
 `reserve`.
 destruct: Not safe, must not be  trusted.


Low level memory management isn't easy.
Bugs like these wouldn't exist if D had decent smart pointer 
implementations in its standard library. You seem knowledgeable 
enough, care to lend a hand? ; )
Feb 13 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Sat, 14 Feb 2015 00:57:33 +0000, weaselcat wrote:


 On Friday, 13 February 2015 at 22:55:27 UTC, anonymous wrote:

 On Thursday, 12 February 2015 at 23:52:41 UTC, Foo wrote:

 This is something I've done recently.
 Would be glad if my code will help you: https://github.com/Dgame/m3
 Especially the m3.d module could be useful for you.


 /* Class and Struct */

 emplace: You can't assume zero-initialization for structs. destruct: Is
 not memory-safe, and must not be marked  trusted.

 /* Array */

 make: You can't assume zero-initialization. T.sizeof is not the size of
 an element.
 reserve: Not safe (you're freeing the old memory), must not be
  trusted.
 append: T.sizeof is not the size of an element. You're multiplying
 twice with T.sizeof; in `append`, and also in `reserve`.
 destruct: Not safe, must not be  trusted.


=20
 Low level memory management isn't easy.
 Bugs like these wouldn't exist if D had decent smart pointer
 implementations in its standard library. You seem knowledgeable enough,
 care to lend a hand? ; )


how "smart pointers" can help here? they don't magically hiding all the=20
low-level mechanics.=
Feb 13 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 14 February 2015 at 06:38:19 UTC, ketmar wrote:

 On Sat, 14 Feb 2015 00:57:33 +0000, weaselcat wrote:


 On Friday, 13 February 2015 at 22:55:27 UTC, anonymous wrote:

 On Thursday, 12 February 2015 at 23:52:41 UTC, Foo wrote:

 This is something I've done recently.
 Would be glad if my code will help you: 
 https://github.com/Dgame/m3
 Especially the m3.d module could be useful for you.


 /* Class and Struct */

 emplace: You can't assume zero-initialization for structs. 
 destruct: Is
 not memory-safe, and must not be marked  trusted.

 /* Array */

 make: You can't assume zero-initialization. T.sizeof is not 
 the size of
 an element.
 reserve: Not safe (you're freeing the old memory), must not be
  trusted.
 append: T.sizeof is not the size of an element. You're 
 multiplying
 twice with T.sizeof; in `append`, and also in `reserve`.
 destruct: Not safe, must not be  trusted.


 
 Low level memory management isn't easy.
 Bugs like these wouldn't exist if D had decent smart pointer
 implementations in its standard library. You seem 
 knowledgeable enough,
 care to lend a hand? ; )


 how "smart pointers" can help here? they don't magically hiding 
 all the
 low-level mechanics.


The file he was referring to was a smart pointer implementation.
Feb 13 2015
parent ketmar <ketmar ketmar.no-ip.org> writes:
On Sat, 14 Feb 2015 07:59:51 +0000, weaselcat wrote:


 how "smart pointers" can help here? they don't magically hiding all the
 low-level mechanics.


=20
 The file he was referring to was a smart pointer implementation.


*one* of the files.=
Feb 14 2015
prev sibling parent reply "Foo" <Foo test.de> writes:
On Friday, 13 February 2015 at 22:55:27 UTC, anonymous wrote:

 On Thursday, 12 February 2015 at 23:52:41 UTC, Foo wrote:

 This is something I've done recently.
 Would be glad if my code will help you:
 https://github.com/Dgame/m3
 Especially the m3.d module could be useful for you.


 /* Class and Struct */

 emplace: You can't assume zero-initialization for structs.
 destruct: Is not memory-safe, and must not be marked  trusted.

 /* Array */

 make: You can't assume zero-initialization. T.sizeof is not the 
 size of an element.
 reserve: Not safe (you're freeing the old memory), must not be 
  trusted.
 append: T.sizeof is not the size of an element. You're 
 multiplying twice with T.sizeof; in `append`, and also in 
 `reserve`.
 destruct: Not safe, must not be  trusted.


Finally, I tried to take your criticism objectively and, as far 
as I can tell, I resolved the issues. Is there still any 
objections or misconduct?
Feb 21 2015
parent reply "anonymous" <anonymous example.com> writes:
On Saturday, 21 February 2015 at 13:41:41 UTC, Foo wrote:

 Finally, I tried to take your criticism objectively and, as far 
 as I can tell, I resolved the issues. Is there still any 
 objections or misconduct?


Nice. I think you fixed everything I had pointed out.
Feb 21 2015
parent "Foo" <Foo test.de> writes:
On Saturday, 21 February 2015 at 14:39:47 UTC, anonymous wrote:

 On Saturday, 21 February 2015 at 13:41:41 UTC, Foo wrote:

 Finally, I tried to take your criticism objectively and, as 
 far as I can tell, I resolved the issues. Is there still any 
 objections or misconduct?


 Nice. I think you fixed everything I had pointed out.


Yeah, I hope so too. Sorry for my annoyed behaviour, that wasn't 
my week and you where the the straw to break the camel's back. ;)

I hope I have the phrase translated correctly. :D
Feb 21 2015
prev sibling next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Thursday, 12 February 2015 at 23:27:51 UTC, Kitt wrote:

 The Exception obviously uses the GC, and would need to be 
 replaced with a printf or something; however, emplace doesn't 
 have a  nogc replacement that I know of. What I'd like to know, 
 from people much more knowledgeable about the ins and outs of 
 D, is what the "Best" or "Correct" way to allocate and 
 deallocate within  nogc sections?


This issue, as well as


 PS: Tangentially related, what hashing algorithm does D use? 
 I've seen people suggest using typeid(object).toHash(&object); 
 however, toHash isn't  nogc, so I need to write my own Hashing 
 function. For the sake of consistency and peace of mind, I'd 
 really like to match mine as closely to Ds internal one as 
 possible.


this one, are related to the loss of guarantee attributes when 
using TypeInfo methods, most recently discussed in this 
thread[1]. It is essentially a bug, and a work in progress.

The correct way to handle OOM in D is to call 
core.exception.onOutOfMemoryError:

---
if (auto p = malloc(...))
     /* use p */
else
     onOutOfMemoryError(); // Throws OutOfMemoryError without 
allocating memory
---

[1] 
http://forum.dlang.org/post/krccldftrbbczahasvbi forum.dlang.org
Feb 12 2015
parent "Kitt" <NekoAssassin gmail.com> writes:
On Friday, 13 February 2015 at 05:56:37 UTC, Jakob Ovrum wrote:

 This issue, ... [is] related to the loss of guarantee 
 attributes when using TypeInfo methods, most recently discussed 
 in this thread[1]. It is essentially a bug, and a work in 
 progress.


I see. So, in theory, after some patching emplace(), 
onOutOfMemoryError() and
toHash() should all work in  nogc sections?

If this is the case, Foo's m3.d module looks rather helpful, and
should tide me over until this fix is released; though, I'll be
honest when I say that I don't know how idiomatic or "correct"
Foo's implementation is.
Feb 13 2015
prev sibling parent "Benjamin Thaut" <code benjamin-thaut.de> writes:
Hi,

you can also take a look at my implementation:

https://github.com/Ingrater/druntime/blob/master/src/core/allocator.d

Look at AllocatorNew and AllocatorDelete

Especially important is, that you correctly handle the 
constructor throwing an exception. You have to catch that 
exception in your new function, release the memory and rethrow.

Kind Regards
Benjamin Thaut
Feb 16 2015