www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Advice wanted on garbage collection of sockets for c++ programmer

reply John Burton <john.burton jbmail.com> writes:
I'm coming from a C++ background so I'm not too used to garbage 
collection and it's implications. I have a function that creates 
a std.socket.Socket using new and connects to a tcp server, and 
writes some stuff to it. I then explicitly close the socket, and 
the socket object goes out of scope.

I assume that I do need to explicitly call close on the socket as 
there is no deterministic destructor for class objects. I further 
assume that the runtime will garbage collect any memory allocated 
to the socket object at a later time.

Am I doing this right with GC? In C++ I'd ensure that the Socket 
class had a destructor that closed the socket and I'd also assume 
that once it went out of scope there was no memory left 
allocated. In D am I right to assume I need to manually close the 
socket but there is no need to worry about the memory?

Now the issue is that I now need to call this function more than 
once every second. I worry that it will create large amounts of 
uncollected "garbage" which will eventually lead to problems.

Am I doing this right? Or is there a better way to do this in D?

Thanks.
Jun 27
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
 I'm coming from a C++ background so I'm not too used to garbage 
 collection and it's implications. I have a function that 
 creates a std.socket.Socket using new and connects to a tcp 
 server, and writes some stuff to it. I then explicitly close 
 the socket, and the socket object goes out of scope.

 I assume that I do need to explicitly call close on the socket 
 as there is no deterministic destructor for class objects. I 
 further assume that the runtime will garbage collect any memory 
 allocated to the socket object at a later time.

 Am I doing this right with GC? In C++ I'd ensure that the 
 Socket class had a destructor that closed the socket and I'd 
 also assume that once it went out of scope there was no memory 
 left allocated. In D am I right to assume I need to manually 
 close the socket but there is no need to worry about the memory?
Yes. from the documentation
Immediately drop any connections and release socket resources. 
Calling shutdown before close is recommended for 
connection-oriented sockets. The Socket object is no longer 
usable after close. Calling shutdown() before this is 
recommended for connection-oriented sockets.
The GC (if you use it) will pick up the garbage for you.
 Now the issue is that I now need to call this function more 
 than once every second. I worry that it will create large 
 amounts of uncollected "garbage" which will eventually lead to 
 problems.
Have a look at automem (https://github.com/atilaneves/automem/)
 Am I doing this right? Or is there a better way to do this in D?

 Thanks.
There is also vibe.d if you are looking for more networking stuff. https://github.com/rejectedsoftware/vibe.d Best of luck Nic
Jun 27
prev sibling next sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Tuesday, June 27, 2017 09:54:19 John Burton via Digitalmars-d-learn 
wrote:
 I'm coming from a C++ background so I'm not too used to garbage
 collection and it's implications. I have a function that creates
 a std.socket.Socket using new and connects to a tcp server, and
 writes some stuff to it. I then explicitly close the socket, and
 the socket object goes out of scope.

 I assume that I do need to explicitly call close on the socket as
 there is no deterministic destructor for class objects. I further
 assume that the runtime will garbage collect any memory allocated
 to the socket object at a later time.

 Am I doing this right with GC? In C++ I'd ensure that the Socket
 class had a destructor that closed the socket and I'd also assume
 that once it went out of scope there was no memory left
 allocated. In D am I right to assume I need to manually close the
 socket but there is no need to worry about the memory?

 Now the issue is that I now need to call this function more than
 once every second. I worry that it will create large amounts of
 uncollected "garbage" which will eventually lead to problems.

 Am I doing this right? Or is there a better way to do this in D?
Arguably, std.socket should have used structs instead of classes for sockets for precisely this reason (though there are some advantages in using inheritance with sockets). But yes, calling close manually is the correct thing to do. Relying on the GC to call a destructor/finalizer is error-prone. There is no guarantee that the memory will ever be freed (e.g. the runtime could choose to not bother doing cleanup on shutdown), and even if the GC does collect it, there are no guarantees about how soon it will do so. However, if you keep allocating memory with the GC, then over time, the GC will collect GC-allocated memory that isn't currently being used so that it can reuse the memory. So, you really don't need to worry about the memory unless it becomes a bottleneck. It will be collected and reused, not leaked. And if in C++, you would be newing up a socket object each time and then deleting it rather than having the socket on the stack, then that performance could actually be worse than using the GC and having it collect the memory when it determines that it needs more memory. If you do find that it becomes a bottleneck to be allocating all of these sockets, then you can look into using std.typecons.scoped, which would allocate a class on the stack, but that really only works if you're just dealing with a local variable, and in general, I wouldn't worry about anything like that unless the heap allocations are proving to be a performance problem - particularly because you need to be very careful when using it to avoid screwing up and having memory corruption, because you used an object after it was freed. - Jonathan M Davis
Jun 27
next sibling parent reply John Burton <john.burton jbmail.com> writes:
On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote:
 On Tuesday, June 27, 2017 09:54:19 John Burton via 
 Digitalmars-d-learn wrote:
 I'm coming from a C++ background so I'm not too used to 
 garbage collection and it's implications. I have a function 
 that creates a std.socket.Socket using new and connects to a 
 tcp server, and writes some stuff to it. I then explicitly 
 close the socket, and the socket object goes out of scope.
Thank you for the advice everyone. The hardest part about learning D isn't the language, or how to program, it's unlearning what you know from C++ and learning the proper way to do things in D. I've tried D several times before and eventually stopped when I get to the stage of "how do I do this c++ thing in d" proves to be hard. Instead this time, I've started writing D programs as "better C" and then slowly started adding in higher level d features. It's going much better as I'm no longer trying so hard to write bad C++ in D :)
Jun 27
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Tuesday, 27 June 2017 at 11:43:27 UTC, John Burton wrote:
 On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis 
 wrote:
 On Tuesday, June 27, 2017 09:54:19 John Burton via 
 Digitalmars-d-learn wrote:
 I'm coming from a C++ background so I'm not too used to 
 garbage collection and it's implications. I have a function 
 that creates a std.socket.Socket using new and connects to a 
 tcp server, and writes some stuff to it. I then explicitly 
 close the socket, and the socket object goes out of scope.
Thank you for the advice everyone. The hardest part about learning D isn't the language, or how to program, it's unlearning what you know from C++ and learning the proper way to do things in D. I've tried D several times before and eventually stopped when I get to the stage of "how do I do this c++ thing in d" proves to be hard. Instead this time, I've started writing D programs as "better C" and then slowly started adding in higher level d features. It's going much better as I'm no longer trying so hard to write bad C++ in D :)
Heh, there's no reason you can't, you'll just cause a few head scratches when you post code while we figure out what it does. Idiomatic D is rather different to C++ and is centered around ranges + algorithms, but you can write code in pretty much any style. Some of the advanced metaprogramming takes a bit to get your head around. If you do get stuck do post a question here, we're more than happy to help.
Jun 27
prev sibling parent bauss <jj_1337 live.dk> writes:
On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote:
 On Tuesday, June 27, 2017 09:54:19 John Burton via 
 Digitalmars-d-learn wrote:
 [...]
Arguably, std.socket should have used structs instead of classes for sockets for precisely this reason (though there are some advantages in using inheritance with sockets). But yes, calling close manually is the correct thing to do. Relying on the GC to call a destructor/finalizer is error-prone. There is no guarantee that the memory will ever be freed (e.g. the runtime could choose to not bother doing cleanup on shutdown), and even if the GC does collect it, there are no guarantees about how soon it will do so. However, if you keep allocating memory with the GC, then over time, the GC will collect GC-allocated memory that isn't currently being used so that it can reuse the memory. So, you really don't need to worry about the memory unless it becomes a bottleneck. It will be collected and reused, not leaked. [...]
I agree with that it should have been structs. The inheritance issue could be fixed by having a private member of the struct in a class, that way there could still be a class wrapper around it.
Jun 27
prev sibling next sibling parent Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
 Now the issue is that I now need to call this function more 
 than once every second. I worry that it will create large 
 amounts of uncollected "garbage" which will eventually lead to 
 problems.
Since nobody has mentioned Allocator, yet: As you seem to know the lifetime of the socket statically, you can just use std.experimental.allocator.{make,dispose} [1]. With regards to reusing the memory: Simply use a freelist allocator [2]. [1] https://dlang.org/phobos/std_experimental_allocator.html#.make [2] https://dlang.org/phobos/std_experimental_allocator_building_blocks_free_list.html
Jun 27
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
 I assume that I do need to explicitly call close on the socket 
 as there is no deterministic destructor for class objects.
Yes.
 I further assume that the runtime will garbage collect any 
 memory allocated to the socket object at a later time.
Most probably.
 Am I doing this right with GC? In C++ I'd ensure that the 
 Socket class had a destructor that closed the socket and I'd 
 also assume that once it went out of scope there was no memory 
 left allocated. In D am I right to assume I need to manually 
 close the socket but there is no need to worry about the memory?
Yes. You can also call a destructor manually with destroy(obj); This avoids having a forest of 'close' methods, who are duplicates of the destructor in spirit, and let's you use destructors like you are accustomed in C++. Generally, it is dangerous to let the GC handle resource release: https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors
 Now the issue is that I now need to call this function more 
 than once every second. I worry that it will create large 
 amounts of uncollected "garbage" which will eventually lead to 
 problems.
If this is a problem you can create it on the stack with std.typecons.scoped
 Am I doing this right? Or is there a better way to do this in D?
What you are doing it OK.
Jun 27
next sibling parent cym13 <cpicard openmailbox.org> writes:
On Tuesday, 27 June 2017 at 12:29:03 UTC, Guillaume Piolat wrote:
 [...]
Hmm... Isn't it possible to just allocate the object/a pool of objects outside the loop and reuse it? Everybody's proposing other means of allocations which is nice, but I wonder why there is such a need for reallocations in the first place. Of course the specifics depend on the implementation, maybe it isn't as straightforward. It's just that no allocation is faster than reallocation no matter what reallocation method is used.
Jun 27
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 8:29 AM, Guillaume Piolat wrote:
 On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
 Am I doing this right with GC? In C++ I'd ensure that the Socket class 
 had a destructor that closed the socket and I'd also assume that once 
 it went out of scope there was no memory left allocated. In D am I 
 right to assume I need to manually close the socket but there is no 
 need to worry about the memory?
Yes. You can also call a destructor manually with destroy(obj); This avoids having a forest of 'close' methods, who are duplicates of the destructor in spirit, and let's you use destructors like you are accustomed in C++. Generally, it is dangerous to let the GC handle resource release: https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors
This is the best to read, it's very tricky for people coming from other languages I think to understand the rules around the GC destructor. I think you have gotten sound advice from many people, but I wanted to chime in as someone who used the GC to clean up sockets and file handles in a long-running program. Even with all these warnings, it does seem to work. My advice is to close the socket in the destructor if still valid, and then manually close it also. This way you don't leak resources if you miss a close call. The fundamental problem with closing sockets and other resources with the GC is that the resource management issues and rules for memory are vastly different than those of other resources. The OS can give you vast sums of memory, but will likely limit your ability to keep sockets open. It also keeps the other end potentially locked up. If you try to allocate a new socket and run out of sockets, there is no "socket collection cycle". This is why closing manually is advised. But I would use a close method, and not destroy(obj). The reason is because often times, you have wrapper types around your socket type, and just one extra level of indirection means the destructor cannot be used to clean up the socket (you are not allowed to access GC-allocated resources in a destructor). Using std.typecons.RefCounted also can help to ensure destruction of a socket when it's no longer used. But like I said, it's not always possible to use a destructor of a higher level object to clean up a socket. -Steve
Jun 27
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer 
wrote:
 But I would use a close method, and not destroy(obj). The 
 reason is because often times, you have wrapper types around 
 your socket type, and just one extra level of indirection means 
 the destructor cannot be used to clean up the socket (you are 
 not allowed to access GC-allocated resources in a destructor).
All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap. My reasoning went with the following: 1 - "I should have close() methods so that I don't rely on the GC, and the GC is calling ~this from the wrong thread etc, not everything can be released in this context (eg: OpenGL objects should be released from one thread only). Close methods will call close methods of "owned" objects." 2 - "Uh, oh. Refcounted and Unique and Scoped all use .destroy, so destructors should call close() methods. To avoid double release, close should handle being called several times. The GC will close what I forgot!" 3 - "Actually there is no order of destruction when called by the GC. So I can't release owned objects when called by the GC. Better do nothing when close() is called by the GC, let's detect this. 4 - close() is now identical with ~this (at least for classes). Let's remove the close() method. Crash when a resource is freed by the GC. That's how the GC-proof resource class came to existence, after many destruction bugs, and it let's you use the GC as a detector for non-deterministic destruction. I miss it in nogc :) https://p0nce.github.io/d-idioms/#GC-proof-resource-class Remember years ago when Alexandrescu suggested the GC shouldn't call heap destructors? That's what we get for having said no at the time.
Jun 27
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 9:25 AM, Guillaume Piolat wrote:
 On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote:
 But I would use a close method, and not destroy(obj). The reason is 
 because often times, you have wrapper types around your socket type, 
 and just one extra level of indirection means the destructor cannot be 
 used to clean up the socket (you are not allowed to access 
 GC-allocated resources in a destructor).
All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap.
Yes, Tango solved this by having a separate "finalize()" method. I wish we had something like this.
 
 My reasoning went with the following:
 
 1 - "I should have close() methods so that I don't rely on the GC, and 
 the GC is calling ~this from the wrong thread etc, not everything can be 
 released in this context (eg: OpenGL objects should be released from one 
 thread only). Close methods will call close methods of "owned" objects."
I hadn't realized this, that is a good point. However, not all resources are restricted this much.
 That's how the GC-proof resource class came to existence, after many 
 destruction bugs, and it let's you use the GC as a detector for 
 non-deterministic destruction. I miss it in  nogc :)
 
 https://p0nce.github.io/d-idioms/#GC-proof-resource-class
There are definitely better ways to do this than trying to allocate and catching an error. The GC knows the GC is running, use that mechanism instead :)
 Remember years ago when Alexandrescu suggested the GC shouldn't call 
 heap destructors? That's what we get for having said no at the time.
No, the issue is that there isn't a separate call for destruction and GC. I think we should add this. The GC still needs to call something to clean up non-memory resources, but having a call to use when cleaning up deterministically can make resource management more efficient (even memory resource management). -Steve -Steve
Jun 27
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 11:24 AM, Steven Schveighoffer wrote:
 On 6/27/17 9:25 AM, Guillaume Piolat wrote:
 That's how the GC-proof resource class came to existence, after many 
 destruction bugs, and it let's you use the GC as a detector for 
 non-deterministic destruction. I miss it in  nogc :)

 https://p0nce.github.io/d-idioms/#GC-proof-resource-class
There are definitely better ways to do this than trying to allocate and catching an error. The GC knows the GC is running, use that mechanism instead :)
https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189 https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150 Apparently that info is limited to the core.memory package. But it's extern(C), so declaring the prototype should work. Should be much more efficient than allocating a byte (which I see you don't delete if it works), and catching an error. -Steve
Jun 27
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 12:16 PM, Steven Schveighoffer wrote:
 On 6/27/17 11:24 AM, Steven Schveighoffer wrote:
 On 6/27/17 9:25 AM, Guillaume Piolat wrote:
 That's how the GC-proof resource class came to existence, after many 
 destruction bugs, and it let's you use the GC as a detector for 
 non-deterministic destruction. I miss it in  nogc :)

 https://p0nce.github.io/d-idioms/#GC-proof-resource-class
There are definitely better ways to do this than trying to allocate and catching an error. The GC knows the GC is running, use that mechanism instead :)
https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189 https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150 Apparently that info is limited to the core.memory package. But it's extern(C), so declaring the prototype should work. Should be much more efficient than allocating a byte (which I see you don't delete if it works), and catching an error.
Just added this enhancement to publicize this function: https://issues.dlang.org/show_bug.cgi?id=17563 -Steve
Jun 27
prev sibling next sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer 
wrote:
 On 6/27/17 9:25 AM, Guillaume Piolat wrote:
 On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer 
 wrote:
 But I would use a close method, and not destroy(obj). The 
 reason is because often times, you have wrapper types around 
 your socket type, and just one extra level of indirection 
 means the destructor cannot be used to clean up the socket 
 (you are not allowed to access GC-allocated resources in a 
 destructor).
All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap.
Yes, Tango solved this by having a separate "finalize()" method. I wish we had something like this.
Well, technically speaking the `~this` for D classes *is* a finalizer that you may optionally manually call (e.g. via destroy). It would be nice, though, to change class `~this` into a destructor and move the finalization into an extra method like `finalize`. Write a DIP?
Jun 27
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/17 2:04 PM, Moritz Maxeiner wrote:
 On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer wrote:
 On 6/27/17 9:25 AM, Guillaume Piolat wrote:
 On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote:
 But I would use a close method, and not destroy(obj). The reason is 
 because often times, you have wrapper types around your socket type, 
 and just one extra level of indirection means the destructor cannot 
 be used to clean up the socket (you are not allowed to access 
 GC-allocated resources in a destructor).
All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap.
Yes, Tango solved this by having a separate "finalize()" method. I wish we had something like this.
Well, technically speaking the `~this` for D classes *is* a finalizer that you may optionally manually call (e.g. via destroy). It would be nice, though, to change class `~this` into a destructor and move the finalization into an extra method like `finalize`. Write a DIP?
I don't have enough motivation to do this. But if we do anything, we should look at what Tango has done. It may not work exactly for druntime because of existing code, but it obviously didn't need compiler changes to work. -Steve
Jun 27
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote:
 Well, technically speaking the `~this` for D classes *is* a 
 finalizer that you may optionally manually call (e.g. via 
 destroy).
 It would be nice, though, to change class `~this` into a 
 destructor and move the finalization into an extra method like 
 `finalize`. Write a DIP?
The only sane way out is to prevent the GC from calling constructors.
Jun 27
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Tuesday, 27 June 2017 at 23:42:38 UTC, Guillaume Piolat wrote:
 On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote:
 Well, technically speaking the `~this` for D classes *is* a 
 finalizer that you may optionally manually call (e.g. via 
 destroy).
 It would be nice, though, to change class `~this` into a 
 destructor and move the finalization into an extra method like 
 `finalize`. Write a DIP?
The only sane way out is to prevent the GC from calling constructors.
Do you mean destructors? If so, that's what I meant with the "technically". If the may GC call it (automatically), it's a finalizer, not a destructor (in the usual sense, anyway). - Replace calls by the GC to `~this` with calls to `finalize` (or invent some cool other shortened name for the latter) - Reserve `~this` for being called by deterministic lifetime management (std.experimental.allocator.dispose, object.destroy, RefCounted, Unique, etc.)
Jun 27
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:
 Do you mean destructors?
Yes.
 - Replace calls by the GC to `~this` with calls to `finalize` 
 (or invent some cool other shortened name for the latter)
My point is that in such a "finalize()" function the only sane things to do is to crash if the resource wasn't freed already. Why so? See my texture example in: https://forum.dlang.org/post/kliasvljzwjhzuzibuae forum.dlang.org More simply: If A needs an alive B to be destroyed, then neither A-owns-B or B-owns-A can guarantee the ordering under GC destruction. Furthermore, this is viral. Whoever owns A needs an alive B to be destroyed.
 - Reserve `~this` for being called by deterministic lifetime 
 management (std.experimental.allocator.dispose, object.destroy, 
 RefCounted, Unique, etc.)
That's precisely what the GC-proof-resource-class allows, by preventing defective, non-composable usages of ~this.
Jun 27
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat 
wrote:
 On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:
 - Replace calls by the GC to `~this` with calls to `finalize` 
 (or invent some cool other shortened name for the latter)
My point is that in such a "finalize()" function the only sane things to do is to crash if the resource wasn't freed already. Why so? [...]
Not every class can't be finalized, so it might make sense for finalization to remain an available option. What I think reasonable is treating destruction and finalization as two distinct things. This could look like the following: --- // Does not refer to other GC memory, can be finalized class File { private: int fd; public: this() { fd = open(...); } ~this() { close(fd); } // If the GC collects a File object that hasn't been destroyed yet, it will call this finalizer, // which *manually* calls the destructor (because it's safe to do so in this case) // A File object can still be manually `destroy`ed beforehand, in which case this particular finalizer is still safe, since an object won't be destroyed twice finalize() { destroy(this); } // Or maybe prettier: alias finalize = ~this; } class Foo { private: File f; public: this() { f = new File(); } // Make the process crash on finalization of an undestroyed Foo object disable finalize(); } ---
 - Reserve `~this` for being called by deterministic lifetime 
 management (std.experimental.allocator.dispose, 
 object.destroy, RefCounted, Unique, etc.)
That's precisely what the GC-proof-resource-class allows, by preventing defective, non-composable usages of ~this.
The GC proof resource class idiom is a necessary hack, but it *is* a hack with its own limitations (e.g. depending on Error being catchable).
Jun 27
parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via Digitalmars-d-learn 
wrote:
 On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat

 wrote:
 On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:
 - Replace calls by the GC to `~this` with calls to `finalize`
 (or invent some cool other shortened name for the latter)
My point is that in such a "finalize()" function the only sane things to do is to crash if the resource wasn't freed already. Why so? [...]
Not every class can't be finalized, so it might make sense for finalization to remain an available option.
There are definitely cases where finalizers make sense. Case in point: if you have a socket class, it makes perfect sense for it to have a finalizer. Yes, it's better to close it manually, but it will work just fine for the GC to close it when finalizing the class object so long as you don't use so many sockets that you run out before the GC collects them. So, having a finalizer is a good backup to ensure that the socket resource doesn't leak. - Jonathan M Davis
Jun 27
next sibling parent Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
wrote:
 On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via 
 Digitalmars-d-learn wrote:
 Not every class can't be finalized, so it might make sense for 
 finalization to remain an available option.
There are definitely cases where finalizers make sense. Case in point: if you have a socket class, it makes perfect sense for it to have a finalizer. Yes, it's better to close it manually, but it will work just fine for the GC to close it when finalizing the class object so long as you don't use so many sockets that you run out before the GC collects them. So, having a finalizer is a good backup to ensure that the socket resource doesn't leak.
Yes, I think that's like the File class I presented (at least on Posix). But on the other hand, since there are also cases in which finalization of an alive (==undestroyed) object are unacceptable (e.g. when it requires other GC managed objects to be alive), splitting them off into two separate methods and allowing to forbid finalization seems sensible to me.
Jun 27
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
wrote:
 There are definitely cases where finalizers make sense. Case in 
 point: if you have a socket class, it makes perfect sense for 
 it to have a finalizer. Yes, it's better to close it manually, 
 but it will work just fine for the GC to close it when 
 finalizing the class object so long as you don't use so many 
 sockets that you run out before the GC collects them. So, 
 having a finalizer is a good backup to ensure that the socket 
 resource doesn't leak.

 - Jonathan M Davis
So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example.
Jun 28
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:
 On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis 
 wrote:
 There are definitely cases where finalizers make sense. Case 
 in point: if you have a socket class, it makes perfect sense 
 for it to have a finalizer. Yes, it's better to close it 
 manually, but it will work just fine for the GC to close it 
 when finalizing the class object so long as you don't use so 
 many sockets that you run out before the GC collects them. So, 
 having a finalizer is a good backup to ensure that the socket 
 resource doesn't leak.

 - Jonathan M Davis
So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example.
And I'm gonna rant a little bit more. Deterministic destruction is a _solved_ problem in C++, and a number of users to convert are now coming from C++. We should: 1. be honest and tell things as they are: it's more complicated than in C++, but also liberating when you know to juggle between GC and deterministic 2. Avoid making bogus suggestions which don't always work, such as close() methods, releasing resource with GC, . They only work for _some_ resources. 3. Suggest a systematic, always working, workaround (or language change such as "GC doesn't call ~this). C++ users have no difficulty having an object graph with detailed ownership schemes, that's what they do day in day out. It's important to have a _simple_ story about resource release, else we will talk about "GC" every day, and hearing about "GC" is bad for marketing.
Jun 28
parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 09:22:07 UTC, Guillaume Piolat 
wrote:
 Deterministic destruction is a _solved_ problem in C++, and a 
 number of users to convert are now coming from C++.
It is also in D, as long as you don't use the GC (which is inherently non-deterministic).
   3. Suggest a systematic, always working, workaround (or 
 language change such as "GC doesn't call ~this). C++ users have 
 no difficulty having an object graph with detailed ownership 
 schemes, that's what they do day in day out.

 It's important to have a _simple_ story about resource release, 
 else we will talk about "GC" every day, and hearing about "GC" 
 is bad for marketing.
Requirement: Do not allocate using the GC Option 1) Use structs with ` disable this`, ` disable this(this)`, and a destructor that checks whether the resource reference is != invalid resource reference before trying to release. Option 2) Use classes with simple constructor/destructor layout. If you want to integrate a check that the requirement holds (with either class or struct), put `if (gc_inFinalizer) throw SomeError` into the class/struct destructor
Jun 28
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner wrote:
 Requirement: Do not allocate using the GC
 Option 1) Use structs with ` disable this`, ` disable 
 this(this)`, and a destructor that checks whether the resource 
 reference is != invalid resource reference before trying to 
 release.
 Option 2) Use classes with simple constructor/destructor layout.
 If you want to integrate a check that the requirement holds 
 (with either class or struct), put `if (gc_inFinalizer) throw 
 SomeError` into the class/struct destructor
I don't get it. It's completely possible to use the full power of GC and be deterministic. I'm out of this very _confused_ discussion.
Jun 28
parent Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 12:33:24 UTC, Guillaume Piolat 
wrote:
 On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner 
 wrote:
 Requirement: Do not allocate using the GC
 Option 1) Use structs with ` disable this`, ` disable 
 this(this)`, and a destructor that checks whether the resource 
 reference is != invalid resource reference before trying to 
 release.
 Option 2) Use classes with simple constructor/destructor 
 layout.
 If you want to integrate a check that the requirement holds 
 (with either class or struct), put `if (gc_inFinalizer) throw 
 SomeError` into the class/struct destructor
I don't get it.
You asked for a "_simple_ story about resource release" and the above two options provide that IMHO.
 It's completely possible to use the full power of GC and be 
 deterministic.
Sure, but it's not "simple". With D's GC as the memory allocator you can currently have (1) non-deterministic memory management and non-deterministic object lifetimes: Use `new` or `std.experimental.allocator.make!(std.experimental.allocator.gc_all cator.GCAllocator)` for allocation&construction. The GC will finalize objects with no pointers to them non-deterministically and deallocate the memory after their respective finalization. (2) non-deterministic memory management and deterministic object lifetimes: Construct the objects as with (1), but destruct them by calling `destroy` on them outside of a collection cycle (i.e. when they, and all their members are still considered "live" by the GC). The GC will collect the memory non-deterministically after there are no more pointers to it. Warning: You are responsible for ensuring that the GC will never see an undestroyed object in the collection cycle, because it might try to finalize (call its `~this`) it. You can protect yourself against such finalization attempts by putting `if (gc_inFinalizer) throw Error("Bug! I forgot to destruct!")` in the objects' respective destructors. (3) deterministic memory management and deterministic object lifetimes: Construct the objects as with (1), but destruct&deallocate them by calling `std.experimental.allocator.dispose!(std.experimental.allocator.gc_alloc tor.GCAllocator)`on them outside of a collection cycle. The GC will not do anything for those objects. Same warning as (2) applies.
 I'm out of this very _confused_ discussion.
I'm honestly not sure what's confusing about it.
Jun 28
prev sibling next sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:
 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.
I thought I had (implicitly): B needs to be ` disable finalize`.
Jun 28
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner wrote:
 On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
 wrote:
 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.
I thought I had (implicitly): B needs to be ` disable finalize`.
So in the current language, doesn't exist?
Jun 28
parent Moritz Maxeiner <moritz ucworks.org> writes:
On Wednesday, 28 June 2017 at 12:28:28 UTC, Guillaume Piolat 
wrote:
 On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner 
 wrote:
 On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
 wrote:
 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.
I thought I had (implicitly): B needs to be ` disable finalize`.
So in the current language, doesn't exist?
I thought that was the premise of the discussion? That the GC *currently* invokes the "destructors" as finalizers and that *currently* the only way to avoid that is calling `destroy` on them manually beforehand (since destructors won't be called twice on a single object).
Jun 28
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
wrote:

 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.
What's the issue with DerelictUtil?
Jun 28
parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 June 2017 at 13:02:04 UTC, Mike Parker wrote:
 On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat 
 wrote:

 So far everyone is ignoring my example when A needs B to be 
 destroyed. This happens as soon as you use DerelictUtil for 
 example.
What's the issue with DerelictUtil?
https://forum.dlang.org/post/pmulowxpikjjffkrscct forum.dlang.org Not an issue with DerelictUtil, an issue with the strategy of closing resources in GC with this case. Derelict loaders work-around this by not unloading shared libraries so the GC won't unload shared libs before the resources related to the shared library are freed. https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366
Jun 28
parent Mike Parker <aldacron gmail.com> writes:
On Wednesday, 28 June 2017 at 13:19:37 UTC, Guillaume Piolat 
wrote:

 https://forum.dlang.org/post/pmulowxpikjjffkrscct forum.dlang.org

 Not an issue with DerelictUtil, an issue with the strategy of 
 closing resources in GC with this case.

 Derelict loaders work-around this by not unloading shared 
 libraries so the GC won't unload shared libs before the 
 resources related to the shared library are freed.

 https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366
Yeah, the loaders all used to needlessly unload the shared libraries in a static destructor. The fact that they don't anymore isn't to avoid any GC/destructor issues, but because there's no point in unloading the libraries when the system is going to do it anyway.
Jun 28
prev sibling next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer 
wrote:
 The GC still needs to call something to clean up non-memory 
 resources,
Well, I'd much prefer if the GC would simply not call destructors. Yes, destructor can work for _some_ resources, but because of ordering it also create accidental correctness, or just doesn't work with other resources.
 but having a call to use when cleaning up deterministically can 
 make resource management more efficient (even memory resource 
 management).
Yes. My point is that the destructor is uniquely positionned to be that call. (This would also enables back failing destructors through the use of exceptions.) In current D the problem with allowing the GC to close resources is that there is no ordering of destructors. A sub-tree of the ownership graph is freed by the GC together, but sometime an order is required for releasing. Typically: you want to release a SDL_Texture but the Derelict SharedLib object has been freed already, so you can't call sdl_releaseTexture. It doesn't matter that the texture object held a reference to the ShareLib object since the sub-tree is destroyed together. close() methods don't help with that, the problem is with the GC calling destructors. And if _some_ resource needs an ordering, then this property leak on their owners, so little by little the whole ownership graph need determinism.
Jun 27
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-27 17:24, Steven Schveighoffer wrote:

 Yes, Tango solved this by having a separate "finalize()" method. I wish
 we had something like this.
Not sure if this is the same, but I remember that Tango had a separate method called "dispose" that was called if a class was allocated on the stack, i.e. with the "scope" keyword. -- /Jacob Carlborg
Jun 28
prev sibling next sibling parent Dsby <dushibaiyu yahoo.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
 I'm coming from a C++ background so I'm not too used to garbage 
 collection and it's implications. I have a function that 
 creates a std.socket.Socket using new and connects to a tcp 
 server, and writes some stuff to it. I then explicitly close 
 the socket, and the socket object goes out of scope.

 [...]
May be you can see yu: https://github.com/dushibaiyu/yu
Jun 28
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-27 11:54, John Burton wrote:
 I'm coming from a C++ background so I'm not too used to garbage
 collection and it's implications. I have a function that creates a
 std.socket.Socket using new and connects to a tcp server, and writes
 some stuff to it. I then explicitly close the socket, and the socket
 object goes out of scope.
Yes. You can use "scope (exit)", unless you're already doing so.
 Now the issue is that I now need to call this function more than once
 every second. I worry that it will create large amounts of uncollected
 "garbage" which will eventually lead to problems.
Sounds like you need a connection pool. The vibe.d [1] framework contains connection pools. It's also possible to manually disable the GC for a while and then enable it again. [1] http://vibed.org -- /Jacob Carlborg
Jun 28
prev sibling parent reply John Burton <john.burton jbmail.com> writes:
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
 I'm coming from a C++ background so I'm not too used to garbage 
 collection and it's implications. I have a function that 
 creates a std.socket.Socket using new and connects to a tcp 
 server, and writes some stuff to it. I then explicitly close 
 the socket, and the socket object goes out of scope.
 Am I doing this right? Or is there a better way to do this in D?

 Thanks.
For my use case here, I'm increasingly thinking that just calling the underlying 'C' socket and send calls is better. No need for anything complicated at all for my actual program :)
Jun 28
parent Meta <jared771 gmail.com> writes:
On Wednesday, 28 June 2017 at 15:55:41 UTC, John Burton wrote:
 On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
 I'm coming from a C++ background so I'm not too used to 
 garbage collection and it's implications. I have a function 
 that creates a std.socket.Socket using new and connects to a 
 tcp server, and writes some stuff to it. I then explicitly 
 close the socket, and the socket object goes out of scope.
 Am I doing this right? Or is there a better way to do this in 
 D?

 Thanks.
For my use case here, I'm increasingly thinking that just calling the underlying 'C' socket and send calls is better. No need for anything complicated at all for my actual program :)
One final piece of advice as this thread seemed to have gone off the rails a bit. You can always put a `scope(exit) socket.close();` after you create the socket. This will ensure that the socket will be closed once the scope is exited no matter what... almost, anyway. If an Error is thrown no stack unwinding is done but at this point your program is in an unrecoverable state anyway and you have a lot more to worry about than a socket that hasn't been closed.
Jun 29