www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Managing malloced memory

reply anon <anon anon.com> writes:
I interface to a C library that gives me a malloced object. How 
can I manage that pointer so that it gets freed automatically.
What I've thought of so far:
* scope(exit): not an option because I want to return that memory
* struct wrapper: Doesn't work because if I pass it to another 
function, they also destroy it (sometimes). Also same problem as 
with scope(exit)
* struct wrapped in automem/ refcounted: The struct still leaves 
original scope and calls the destructor
Oct 06 2021
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/6/21 2:06 PM, anon wrote:

 I interface to a C library that gives me a malloced object. How can I 
 manage that pointer so that it gets freed automatically.
 What I've thought of so far:
 * scope(exit): not an option because I want to return that memory
 * struct wrapper: Doesn't work because if I pass it to another function, 
 they also destroy it (sometimes). Also same problem as with scope(exit)
 * struct wrapped in automem/ refcounted: The struct still leaves 
 original scope and calls the destructor


If the memory is the only resource it is consuming, you can use a 
GC-allocated wrapper.

This is how I would do it:

```d
struct GCWrapped(T)
{
    private T *_val;
    this(T* val) { _val = val; }
    ref T get() { return *_val; }
    alias get this; // automatically unwrap
    ~this() { free(_val); _val = null; }
     disable this(this); // disable copying to avoid double-free
}

GCWrapped!T *wrap(T)(T *item) {
   return new GCWrapped!T(item);
}

// usage
auto wrapped = wrap(cFunction());

// use wrapped wherever you need to access a T.
```

You can return this thing and pass it around, and the GC will keep it 
alive until it's not needed. Then on collection, the value is freed.

If this contains a non-memory resource, such as a file or socket, then 
those are much more limited, and you probably want to use deterministic 
destruction instead.

-Steve
Oct 06 2021
next sibling parent anon <anon anon.com> writes:
Thanks for the help.

On Wednesday, 6 October 2021 at 18:29:34 UTC, Steven 
Schveighoffer wrote:

You can return this thing and pass it around, and the GC will 
keep it alive until it's not needed. Then on collection, the 
value is freed.


Is the gc required to call ~this() on the struct? I remember it 
being implementation defined. Probably doesn't matter for my 
usecase, just curious.


Why is it a problem that it calls the dtor?  I thought the whole 
point of refcounting is for the dtor to decrement the refcount, 
and free the malloc'd object only when the refcount has actually 
reached 0.


Yes I'm afraid of double freeing.
Oct 06 2021
prev sibling next sibling parent reply anon <anon anon.com> writes:
Sorry for messed up post, fixed it.

On Wednesday, 6 October 2021 at 18:29:34 UTC, Steven 
Schveighoffer wrote:

You can return this thing and pass it around, and the GC will 
keep it alive until it's not needed. Then on collection, the 
value is freed.


Is the gc required to call ~this() on the struct? I remember it 
being implementation defined. Probably doesn't matter for my 
usecase, just curious.



Why is it a problem that it calls the dtor? I thought the whole 
point of refcounting is for the dtor to decrement the refcount, 
and free the malloc'd object only when the refcount has actually 
reached 0.


Yes I'm afraid of double freeing. How do I pass existing struct 
to refcounted without the already existing copy calling 
destructed on function exit.
Oct 06 2021
next sibling parent anon <anon anon.com> writes:
I found https://dlang.org/library/std/typecons/unique.html , 
which I think solves my problem by disabling copying. Thanks for 
the help.
Oct 06 2021
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/6/21 3:22 PM, anon wrote:

 Sorry for messed up post, fixed it.
 
 On Wednesday, 6 October 2021 at 18:29:34 UTC, Steven Schveighoffer wrote:

 You can return this thing and pass it around, and the GC will keep it 
 alive until it's not needed. Then on collection, the value is freed.


 
 Is the gc required to call ~this() on the struct? I remember it being 
 implementation defined. Probably doesn't matter for my usecase, just 
 curious.


The GC is technically not required to free any blocks ever. But in 
general, it does.

When it does free a struct, as long as you allocated with `new`, it 
should call the dtor.


 Why is it a problem that it calls the dtor? I thought the whole point 
 of refcounting is for the dtor to decrement the refcount, and free the 
 malloc'd object only when the refcount has actually reached 0.


 
 Yes I'm afraid of double freeing. How do I pass existing struct to 
 refcounted without the already existing copy calling destructed on 
 function exit.
 


Just FYI, you should reply to the posts that you quote, or at least copy 
the "X Y wrote" line so people understand the thread. This question was 
written by H.S. Teoh, but I'll respond.

The destructor is called once per copy. This is why disabling copy 
prevents double freeing.

There are cases where the compiler avoids calling the destructor because 
the instance is moved. Such as returning a newly constructed item 
(typically referred to as an "rvalue"), or passing a newly constructed 
item into a parameter. The parameter will be destroyed, but the 
call-site constructed item will not.

e.g.:

```d
struct S
{
    int x;
    ~this() { writeln("destructor called"); }
}

void foo(S s) {

    // destructor for s called here
}

S makeS(int x)
{
    return S(x); // no destructor called here.
}

void main()
{
    foo(S(1)); // no destructor called for this rvalue
    auto s = makeS(1);
    // destructor for s called here.
    foo(makeS(1)); // only one destructor called at the end of foo
}
```

You can also use std.algorithm.move to avoid calling destructors on live 
data. However, the destructor will still be called, it just will be 
called on an S.init value.

-Steve
Oct 07 2021
parent reply anon <anon anon.com> writes:
On Thursday, 7 October 2021 at 11:55:35 UTC, Steven Schveighoffer 
wrote:

 The GC is technically not required to free any blocks ever. But 
 in general, it does.

 When it does free a struct, as long as you allocated with 
 `new`, it should call the dtor.


In practice when I played around with it, destructor always got 
called by GC. But: https://dlang.org/spec/class.html#destructors 
says at point 6:

The garbage collector is not guaranteed to run the destructor 
for all unreferenced objects.


Is it the same for structs or are these destructors guaranteed to 
be called? Would it be suitable to clean up tempfiles with 
GC-managed structs?


 Just FYI, you should reply to the posts that you quote, or at 
 least copy the "X Y wrote" line so people understand the thread.


Alright. If I want to reply to multiple people, should I post 
twice or quote both in the same post?


 The destructor is called once per copy. This is why disabling 
 copy prevents double freeing.

 There are cases where the compiler avoids calling the 
 destructor because the instance is moved. Such as returning a 
 newly constructed item (typically referred to as an "rvalue"), 
 or passing a newly constructed item into a parameter. The 
 parameter will be destroyed, but the call-site constructed item 
 will not.

 e.g.:

 ```d
 struct S
 {
    int x;
    ~this() { writeln("destructor called"); }
 }

 void foo(S s) {

    // destructor for s called here
 }

 S makeS(int x)
 {
    return S(x); // no destructor called here.
 }

 void main()
 {
    foo(S(1)); // no destructor called for this rvalue
    auto s = makeS(1);
    // destructor for s called here.
    foo(makeS(1)); // only one destructor called at the end of 
 foo
 }
 ```


Is there any reference for exactly how these rules apply, or is 
this implementation defined? The 
[specification](https://dlang.org/spec/struct.html#struct-destructor) says that
destructors are called when objects go out of scope. Your examples seem to
suggest that this is untrue in some cases.
Oct 11 2021
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Monday, 11 October 2021 at 10:53:15 UTC, anon wrote:


 S makeS(int x)
 {
    return S(x); // no destructor called here.
 }

 void main()
 {
    foo(S(1)); // no destructor called for this rvalue
    auto s = makeS(1);
    // destructor for s called here.
    foo(makeS(1)); // only one destructor called at the end of 
 foo
 }
 ```


 Is there any reference for exactly how these rules apply, or is 
 this implementation defined? The 
 [specification](https://dlang.org/spec/struct.html#struct-destructor) says
that destructors are called when objects go out of scope. Your examples seem to
suggest that this is untrue in some cases.


For example, in `makeS` the initializer combined with the return 
triggers an optimization (return value optimization, or RVO)) 
that elides a copy of the struct, meaning there's nothing to 
destroy at the end of `makeS`. The destruction will occur in the 
scope into which the instance is moved.

Any time you have a named instance, like `S s = S(1)`, you can 
pretty much guarantee its destructor will be called. An exception 
is when `s` is returned immediately after the initialization, 
then NRVO (named return value optimization) can kick in to elide 
the copy and, therefore, the destruction again happens at the end 
of the scope into which the instance is moved.

Play around with a struct destructor that prints a message and 
you'll get a feel for when destructors are and aren't called. 
Like Steve said, it's once per copy. Sometimes you end up with 
temporaries that are destroyed, sometimes you don't depending on 
compiler optimizations.
Oct 11 2021
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 10/11/21 6:53 AM, anon wrote:

 On Thursday, 7 October 2021 at 11:55:35 UTC, Steven Schveighoffer wrote:

 The GC is technically not required to free any blocks ever. But in 
 general, it does.

 When it does free a struct, as long as you allocated with `new`, it 
 should call the dtor.


 
 In practice when I played around with it, destructor always got called 
 by GC. But: https://dlang.org/spec/class.html#destructors says at point 6:

 The garbage collector is not guaranteed to run the destructor for all 
 unreferenced objects.


 Is it the same for structs or are these destructors guaranteed to be 
 called? Would it be suitable to clean up tempfiles with GC-managed structs?


It's not guaranteed to run the destructor because it's not guaranteed to 
clean up the memory at all. For sure, it will not clean up the memory 
without first running the destructor (except on process termination, 
which will obviously clean up everything without running destructors).

This is par for the course with GCs, they all have fine print that says 
the destructors (finalizers) may not be called. Most of the time it 
means that the memory will not get cleaned up too. But it's technically 
spec-compliant for the GC to not run the dtor and clean up the memory.

Temp files I would say to ensure they are cleaned up synchronously. 
Though my best practice recommendation is to clean up non-memory 
resources using destructors that will leak if you forget to 
synchronously clean them up.


 Just FYI, you should reply to the posts that you quote, or at least 
 copy the "X Y wrote" line so people understand the thread.


 
 Alright. If I want to reply to multiple people, should I post twice or 
 quote both in the same post?


You can do it either way. It just looked from your message like I was 
saying the things that H.S. Teoh did.

For sure, if you want a response from someone, it's good to reply 
directly to their post.


 The destructor is called once per copy. This is why disabling copy 
 prevents double freeing.

 There are cases where the compiler avoids calling the destructor 
 because the instance is moved. Such as returning a newly constructed 
 item (typically referred to as an "rvalue"), or passing a newly 
 constructed item into a parameter. The parameter will be destroyed, 
 but the call-site constructed item will not.





 Is there any reference for exactly how these rules apply, or is this 
 implementation defined? The 
 [specification](https://dlang.org/spec/struct.html#struct-destructor) 
 says that destructors are called when objects go out of scope. Your 
 examples seem to suggest that this is untrue in some cases.


A struct on the heap doesn't go out of scope after the stack frame, 
since it's still on the heap.

Unfortunately, the spec is maintained over history, and historically, 
struct destructors were not run by the GC even when the memory was 
cleaned up. So this terminology is focused on structs that were mostly 
only functional on the stack.

-Steve
Oct 11 2021
prev sibling parent anon <anon anon.com> writes:
On Wednesday, 6 October 2021 at 18:29:34 UTC, Steven 
Schveighoffer wrote:

 ```d
 struct GCWrapped(T)
 {
    private T *_val;
    this(T* val) { _val = val; }
    ref T get() { return *_val; }
    alias get this; // automatically unwrap
    ~this() { free(_val); _val = null; }
     disable this(this); // disable copying to avoid double-free
 }

 GCWrapped!T *wrap(T)(T *item) {
   return new GCWrapped!T(item);
 }

 // usage
 auto wrapped = wrap(cFunction());

 // use wrapped wherever you need to access a T.
 ```


RE:  disable this(this);
I noticed that std.typecons.RefCounted only works on structs if 
you set this line. How is that? Is RefCounted catching an 
exception and working around it, or does the compiler treat 
strcuts like GCWrapped with postblit disabled differently and use 
other operations for them automatically, when it would otherwise 
had copied it. My guess: OpAssign gets converted to a move 
constructor automatically
Oct 12 2021
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Oct 06, 2021 at 06:06:38PM +0000, anon via Digitalmars-d-learn wrote:

 I interface to a C library that gives me a malloced object. How can I manage
 that pointer so that it gets freed automatically.
 What I've thought of so far:


[...]

 * struct wrapped in automem/ refcounted: The struct still leaves
 original scope and calls the destructor


Why is it a problem that it calls the dtor?  I thought the whole point
of refcounting is for the dtor to decrement the refcount, and free the
malloc'd object only when the refcount has actually reached 0.

You do have to be careful about copy ctors, assignment operators, and
such, though, to make sure the refcount stays consistent throughout. And
the refcount should probably be on the heap somewhere (either GC heap or
part of the malloc'd object); you would not want a struct's by-value
semantics to make another copy of the refcount and end up freeing the C
object more than once.


T

-- 
Life is unfair. Ask too much from it, and it may decide you don't deserve what
you have now either.
Oct 06 2021
prev sibling parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Wednesday, 6 October 2021 at 18:06:38 UTC, anon wrote:

 I interface to a C library that gives me a malloced object. How 
 can I manage that pointer so that it gets freed automatically.
 What I've thought of so far:
 * scope(exit): not an option because I want to return that 
 memory
 * struct wrapper: Doesn't work because if I pass it to another 
 function, they also destroy it (sometimes). Also same problem 
 as with scope(exit)
 * struct wrapped in automem/ refcounted: The struct still 
 leaves original scope and calls the destructor


Explain again why scope exit isn't an option
Oct 11 2021
parent reply jfondren <julian.fondren gmail.com> writes:
On Monday, 11 October 2021 at 12:09:07 UTC, Imperatorn wrote:

 On Wednesday, 6 October 2021 at 18:06:38 UTC, anon wrote:

 I interface to a C library that gives me a malloced object. 
 How can I manage that pointer so that it gets freed 
 automatically.
 What I've thought of so far:
 * scope(exit): not an option because I want to return that 
 memory


 Explain again why scope exit isn't an option


The explanation is "I want to return that memory".


```d
int* not_an_option() {
     import core.memory : pureMalloc, pureFree;

     int* p = cast(int*) pureMalloc(int.sizeof);
     scope (exit)
         pureFree(p);
     return p;
}

unittest {
     not_an_option()[0] = 1;
}
```

valgrind: Invalid write of size 4
Oct 11 2021
parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Monday, 11 October 2021 at 12:20:27 UTC, jfondren wrote:

 On Monday, 11 October 2021 at 12:09:07 UTC, Imperatorn wrote:

 On Wednesday, 6 October 2021 at 18:06:38 UTC, anon wrote:

 I interface to a C library that gives me a malloced object. 
 How can I manage that pointer so that it gets freed 
 automatically.
 What I've thought of so far:
 * scope(exit): not an option because I want to return that 
 memory


 Explain again why scope exit isn't an option


 The explanation is "I want to return that memory".


 ```d
 int* not_an_option() {
     import core.memory : pureMalloc, pureFree;

     int* p = cast(int*) pureMalloc(int.sizeof);
     scope (exit)
         pureFree(p);
     return p;
 }

 unittest {
     not_an_option()[0] = 1;
 }
 ```

 valgrind: Invalid write of size 4


Oops
Oct 11 2021