www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Socket handle leak and active handle warning with Vibe-D

reply Selim Ozel <sozel wpi.edu> writes:
I created the simplest possible example as explained by the 
Vibe-D community in [1]. The exact source code of what I run is 
in [2].

On Windows I get a socket handle leak warning on shutdown with 
crtl+c from terminal after running the executable.


 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your 
 browser.
 [00000000(----) INF] Received signal 2. Shutting down.
 Warning: 2 socket handles leaked at driver^ Cshutdown


On Ubuntu 20.04 I get leaking drivers warning with the same 
process.

 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your 
 browser.
 ^C[main(----) INF] Received signal 2. Shutting down.
 Warning (thread: main): leaking eventcore driver because there 
 are still active handles
    FD 6 (streamListen)
    FD 7 (streamListen)
 
 Warning (thread: main): leaking eventcore driver because there 
 are still active handles
    FD 6 (streamListen)
    FD 7 (streamListen)


I really don't know what this is all about but it is at the core 
of my Vibe-D development. So any pointers you might have would be 
very helpful to me.

Thanks in advance.

S

[1] https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d
[2] https://github.com/SelimOzel/vibe_noLeaks
Jan 01 2021
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/1/21 5:07 PM, Selim Ozel wrote:

 I created the simplest possible example as explained by the Vibe-D 
 community in [1]. The exact source code of what I run is in [2].
 
 On Windows I get a socket handle leak warning on shutdown with crtl+c 
 from terminal after running the executable.
 

 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your browser.
 [00000000(----) INF] Received signal 2. Shutting down.
 Warning: 2 socket handles leaked at driver^ Cshutdown


 
 On Ubuntu 20.04 I get leaking drivers warning with the same process.

 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your browser.
 ^C[main(----) INF] Received signal 2. Shutting down.
 Warning (thread: main): leaking eventcore driver because there are 
 still active handles
    FD 6 (streamListen)
    FD 7 (streamListen)

 Warning (thread: main): leaking eventcore driver because there are 
 still active handles
    FD 6 (streamListen)
    FD 7 (streamListen)


 
 I really don't know what this is all about but it is at the core of my 
 Vibe-D development. So any pointers you might have would be very helpful 
 to me.
 
 Thanks in advance.


1. the sockets are leaked for a reason that is pretty obscure -- namely, 
the GC might need to access those sockets as the process is shut down. 
Prior to this, the end result of vibe.d server was frequently a segfault.
2. The reason they are leaking is most likely because you still have a 
listening socket somewhere. I wish it would tell you how that socket was 
allocated, but it doesn't.

To fix, make sure all your listening sockets are closed. In my vibe.d 
app, I have the following:

auto listener = listenHTTP(settings, router);
scope(exit) listener.stopListening();

I also clean up my session store connection (something I had to add 
support for in vibe.d), which was a different source of leaking handles.

I also clean up database connections, which might be cached.

And finally, even with all this, I still get leaking driver messages if 
an HTTP keepalive socket is open.

I really feel like vibe.d should give you the option of not printing 
this message, as most of the time, it's something you can ignore.

-Steve
Jan 01 2021
next sibling parent reply Selim Ozel <sozel wpi.edu> writes:
On Saturday, 2 January 2021 at 00:28:43 UTC, Steven Schveighoffer 
wrote:

 On 1/1/21 5:07 PM, Selim Ozel wrote:

 I created the simplest possible example as explained by the 
 Vibe-D community in [1]. The exact source code of what I run 
 is in [2].
 
 On Windows I get a socket handle leak warning on shutdown with 
 crtl+c from terminal after running the executable.
 

 [...]


 
 On Ubuntu 20.04 I get leaking drivers warning with the same 
 process.

 [...]


 
 I really don't know what this is all about but it is at the 
 core of my Vibe-D development. So any pointers you might have 
 would be very helpful to me.
 
 Thanks in advance.


 1. the sockets are leaked for a reason that is pretty obscure 
 -- namely, the GC might need to access those sockets as the 
 process is shut down. Prior to this, the end result of vibe.d 
 server was frequently a segfault.
 2. The reason they are leaking is most likely because you still 
 have a listening socket somewhere. I wish it would tell you how 
 that socket was allocated, but it doesn't.

 To fix, make sure all your listening sockets are closed. In my 
 vibe.d app, I have the following:

 auto listener = listenHTTP(settings, router);
 scope(exit) listener.stopListening();

 I also clean up my session store connection (something I had to 
 add support for in vibe.d), which was a different source of 
 leaking handles.

 I also clean up database connections, which might be cached.

 And finally, even with all this, I still get leaking driver 
 messages if an HTTP keepalive socket is open.

 I really feel like vibe.d should give you the option of not 
 printing this message, as most of the time, it's something you 
 can ignore.

 -Steve


Hey Steve. Thanks a ton for all the tips. FWIW I am writing down 
my findings below because just maybe they might be helpful for 
someone else later on.

The scope guard seems to have fixed some of the leak complaints. 
Unfortunately there is still a leak after a single connection.

Without connection Windows 10:

 Running .\vibe_noleaks.exe
 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your 
 browser.
 [00000000(----) INF] Received signal 2. Shutting down.
 [main(----) INF] Stopped to listen for HTTP requests on ::1:8080
 [main(----) INF] Stopped to listen for HTTP requests on 
 127.0.0.1:8080


Without connection Ubuntu 20.04:

 Running ./vibe_noleaks
 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your 
 browser.
 ^C[main(----) INF] Received signal 2. Shutting down.
 [main(----) INF] Stopped to listen for HTTP requests on ::1:8080
 [main(----) INF] Stopped to listen for HTTP requests on 
 127.0.0.1:8080


After logging into to 127.0.0.1 for a single time in my browser, 
if I do a ctrl+c it still leaks two socket handles.

With connection Windows 10:

 Running .\vibe_noleaks.exe
 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your 
 browser.
 [00000000(----) INF] Received signal 2. Shutting down.
 [main(----) INF] Stopped to listen for HTTP^ requests on 
 C::1:8080
 [main(----
 ) INFC:\Software\vibe_noLeaks>] Stopped to listen for HTTP 
 requests on 127.0.0.1:8080
 Warning: 2 socket handles leaked at driver shutdown.
 Warning: 2 socket handles leaked at driver shutdown.



I think vibe-d is also leaking more sockets when there is a web 
interface attached (not included in my toy repository). These 
haven't stopped me from developing but they are just things I 
wanted to write down and learn more before building even more 
infrastructure with it.

I might dive into vibe-d codebase at some point too.

Best,
S
Jan 02 2021
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/2/21 12:52 PM, Selim Ozel wrote:


 
 After logging into to 127.0.0.1 for a single time in my browser, if I do 
 a ctrl+c it still leaks two socket handles.
 
 With connection Windows 10:

 Running .\vibe_noleaks.exe
 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your browser.
 [00000000(----) INF] Received signal 2. Shutting down.
 [main(----) INF] Stopped to listen for HTTP^ requests on C::1:8080
 [main(----
 ) INFC:\Software\vibe_noLeaks>] Stopped to listen for HTTP requests on 
 127.0.0.1:8080
 Warning: 2 socket handles leaked at driver shutdown.
 Warning: 2 socket handles leaked at driver shutdown.




This is normal. The server uses keepalive connections, so that in case 
any more requests arrive on the same connection, the initial connection 
setup does not need to be established. Well, at least that is what I 
think is happening.

If you want a few seconds (I think 5 or so), then you won't get these.

It would be good if vibe-d could provide a way to shut down any 
keepalive connections when the server is shutting down.

-Steve
Jan 04 2021
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/4/21 12:17 PM, Steven Schveighoffer wrote:

 If you want a few seconds


*wait* a few seconds

-Steve
Jan 04 2021
prev sibling parent reply Selim Ozel <sozel wpi.edu> writes:
On Monday, 4 January 2021 at 17:17:10 UTC, Steven Schveighoffer 
wrote:

 On 1/2/21 12:52 PM, Selim Ozel wrote:


 
 After logging into to 127.0.0.1 for a single time in my 
 browser, if I do a ctrl+c it still leaks two socket handles.
 
 With connection Windows 10:

 Running .\vibe_noleaks.exe
 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your 
 browser.
 [00000000(----) INF] Received signal 2. Shutting down.
 [main(----) INF] Stopped to listen for HTTP^ requests on 
 C::1:8080
 [main(----
 ) INFC:\Software\vibe_noLeaks>] Stopped to listen for HTTP 
 requests on 127.0.0.1:8080
 Warning: 2 socket handles leaked at driver shutdown.
 Warning: 2 socket handles leaked at driver shutdown.




 This is normal. The server uses keepalive connections, so that 
 in case any more requests arrive on the same connection, the 
 initial connection setup does not need to be established. Well, 
 at least that is what I think is happening.

 If you want a few seconds (I think 5 or so), then you won't get 
 these.

 It would be good if vibe-d could provide a way to shut down any 
 keepalive connections when the server is shutting down.

 -Steve


That's interesting. I actually started to dive deeper into those 
and tried to pinpoint the lines of code that result in additional 
open sockets upon new http connections; although my understanding 
of vibe-d is a bit too low at this point to figure out what's 
exactly happening.

I think from a user perspective having something a bit friendlier 
on warning side would be helpful. Do you have any suggestions in 
mind towards that? I have a bit of time this week and I could 
take a stab at it.

Best,
S
Jan 05 2021
parent aberba <karabutaworld gmail.com> writes:
On Tuesday, 5 January 2021 at 21:12:01 UTC, Selim Ozel wrote:

 On Monday, 4 January 2021 at 17:17:10 UTC, Steven Schveighoffer 
 wrote:

 On 1/2/21 12:52 PM, Selim Ozel wrote:


 
 After logging into to 127.0.0.1 for a single time in my 
 browser, if I do a ctrl+c it still leaks two socket handles.
 
 With connection Windows 10:

 Running .\vibe_noleaks.exe
 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [main(----) INF] Please open http://127.0.0.1:8080/ in your 
 browser.
 [00000000(----) INF] Received signal 2. Shutting down.
 [main(----) INF] Stopped to listen for HTTP^ requests on 
 C::1:8080
 [main(----
 ) INFC:\Software\vibe_noLeaks>] Stopped to listen for HTTP 
 requests on 127.0.0.1:8080
 Warning: 2 socket handles leaked at driver shutdown.
 Warning: 2 socket handles leaked at driver shutdown.




 This is normal. The server uses keepalive connections, so that 
 in case any more requests arrive on the same connection, the 
 initial connection setup does not need to be established. 
 Well, at least that is what I think is happening.

 If you want a few seconds (I think 5 or so), then you won't 
 get these.

 It would be good if vibe-d could provide a way to shut down 
 any keepalive connections when the server is shutting down.

 -Steve


 That's interesting. I actually started to dive deeper into 
 those and tried to pinpoint the lines of code that result in 
 additional open sockets upon new http connections; although my 
 understanding of vibe-d is a bit too low at this point to 
 figure out what's exactly happening.

 I think from a user perspective having something a bit 
 friendlier on warning side would be helpful.


Bausshf built a wrapper around vibe.d sockets called cheetah. 
Haven't used it myself but it looks more abstracted than than 
actual vibe.d code. Can't speak of the quality since I haven't 
used it myself.

Also it's got both a server and client abstraction but only the 
server example is shown. Still not low-level though.

https://github.com/bausshf/cheetah/wiki/Simple-Socket-Server-Example


I've used the very vibe.d example you referenced in the past and 
didn't have any trouble with it. So it could possibly be a bug or 
regression.

If you've have wsl2 installed on Windows, try running on the 
Linux side too.

Do you have any

 suggestions in mind towards that? I have a bit of time this 
 week and I could take a stab at it.

 B
Jan 08 2021
prev sibling parent reply bomat <Tempest_spam gmx.de> writes:
I am still getting this in 2024 and vibe.d 0.9.7:
```
Warning: 1 socket handles leaked at driver shutdown.
```

I was wondering if maybe someone has new info on this...
Jan 13
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Saturday, 13 January 2024 at 20:49:54 UTC, bomat wrote:

 I am still getting this in 2024 and vibe.d 0.9.7:
 ```
 Warning: 1 socket handles leaked at driver shutdown.
 ```

 I was wondering if maybe someone has new info on this...


There should be a version you can enable that tells you where 
that socket handle was allocated. That might give you a further 
clue as to why it's not closed when the system shuts down.

I think the program tells you which version to enable when this 
happens, but if not, let me know and I'll find it.

-Steve
Jan 14
parent reply bomat <Tempest_spam gmx.de> writes:
On Sunday, 14 January 2024 at 20:36:44 UTC, Steven Schveighoffer 
wrote:

 There should be a version you can enable that tells you where 
 that socket handle was allocated. That might give you a further 
 clue as to why it's not closed when the system shuts down.

 I think the program tells you which version to enable when this 
 happens, but if not, let me know and I'll find it.

 -Steve


Thanks for the response, Steve.
Hmmm, not sure if I'm missing something, but this is all the 
output I get from the program:
```
[main(----) INF] Listening for requests on http://[::1]:8080/
[main(----) INF] Listening for requests on http://127.0.0.1:8080/
[00000000(----) INF] Received signal 2. Shutting down.
[main(----) INF] Stopped to listen for HTTP requests on ::1:8080
[main(
----) INF] Stopped to listen for HTTP requests on 127.0.0.1:8080
Warning: 1 socket handles leaked at driver shutdown.
Warning: 1 socket handles leaked at driver shutdown.
```
Unless there's some switch to make it more verbose?
Jan 15
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Monday, 15 January 2024 at 17:24:40 UTC, bomat wrote:

 On Sunday, 14 January 2024 at 20:36:44 UTC, Steven 
 Schveighoffer wrote:

 There should be a version you can enable that tells you where 
 that socket handle was allocated. That might give you a 
 further clue as to why it's not closed when the system shuts 
 down.

 I think the program tells you which version to enable when 
 this happens, but if not, let me know and I'll find it.


 Thanks for the response, Steve.
 Hmmm, not sure if I'm missing something, but this is all the 
 output I get from the program:
 ```
 [main(----) INF] Listening for requests on http://[::1]:8080/
 [main(----) INF] Listening for requests on 
 http://127.0.0.1:8080/
 [00000000(----) INF] Received signal 2. Shutting down.
 [main(----) INF] Stopped to listen for HTTP requests on ::1:8080
 [main(
 ----) INF] Stopped to listen for HTTP requests on 127.0.0.1:8080
 Warning: 1 socket handles leaked at driver shutdown.
 Warning: 1 socket handles leaked at driver shutdown.
 ```
 Unless there's some switch to make it more verbose?


Which driver are you using? In the posix driver, it should 
mention (and use) the debug flag `EventCoreLeakTrace`.

https://github.com/vibe-d/eventcore/blob/7fa0a15fa541c3fcf65640ee332fd3a09c34730c/source/eventcore/drivers/posix/driver.d#L130

I didn't realize this wasn't across the board...

-Steve
Jan 15
parent reply bomat <Tempest_spam gmx.de> writes:
On Monday, 15 January 2024 at 17:45:16 UTC, Steven Schveighoffer 
wrote:

 Which driver are you using? In the posix driver, it should 
 mention (and use) the debug flag `EventCoreLeakTrace`.

 https://github.com/vibe-d/eventcore/blob/7fa0a15fa541c3fcf65640ee332fd3a09c34730c/source/eventcore/drivers/posix/driver.d#L130

 I didn't realize this wasn't across the board...
 -Steve


Sorry, I probably should have mentioned I was on Windows.
For testing it under Linux I commented out the call to 
`connectMongoDB`, since I don't have it installed there - and the 
warning went away.
Interesting, I did not suspect that as the source of the problem 
at all. :P

I'm now looking into how to clean up MongoDB connections 
properly, but don't see anything besides `cleanupConnections()` 
(which I'm already calling without any effect).
Maybe I need to initialize it differently... I'll experiment a 
bit.
Jan 15
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Monday, 15 January 2024 at 18:40:00 UTC, bomat wrote:


 Sorry, I probably should have mentioned I was on Windows.
 For testing it under Linux I commented out the call to 
 `connectMongoDB`, since I don't have it installed there - and 
 the warning went away.
 Interesting, I did not suspect that as the source of the 
 problem at all. :P

 I'm now looking into how to clean up MongoDB connections 
 properly, but don't see anything besides `cleanupConnections()` 
 (which I'm already calling without any effect).
 Maybe I need to initialize it differently... I'll experiment a 
 bit.


You may have to do the same thing I did with redis:

https://github.com/vibe-d/vibe.d/pull/2372

Good luck! I would also say, I don't know why Windows doesn't do 
the same trace info debug thing, except that probably whomever 
added it didn't care about windows.

-Steve
Jan 15
parent bomat <Tempest_spam gmx.de> writes:
On Monday, 15 January 2024 at 22:19:56 UTC, Steven Schveighoffer 
wrote:

 You may have to do the same thing I did with redis:

 https://github.com/vibe-d/vibe.d/pull/2372

 Good luck! I would also say, I don't know why Windows doesn't 
 do the same trace info debug thing, except that probably 
 whomever added it didn't care about windows.


Many thanks for the assistance.
Jan 16
prev sibling next sibling parent Selim Ozel <sozel wpi.edu> writes:
On Friday, 1 January 2021 at 22:07:28 UTC, Selim Ozel wrote:

 I created the simplest possible example as explained by the 
 Vibe-D community in [1]. The exact source code of what I run is 
 in [2].

 On Windows I get a socket handle leak warning on shutdown with 
 crtl+c from terminal after running the executable.


 [...]


 On Ubuntu 20.04 I get leaking drivers warning with the same 
 process.

    [...]


 I really don't know what this is all about but it is at the 
 core of my Vibe-D development. So any pointers you might have 
 would be very helpful to me.

 Thanks in advance.

 S

 [1] 
 https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d
 [2] https://github.com/SelimOzel/vibe_noLeaks


For further reference, I also went through this issue [1]. It 
seems like I am seeing the same behavior as kinexis-uk.

S

[1] https://github.com/vibe-d/vibe.d/issues/2245
Jan 02 2021
prev sibling next sibling parent reply aberba <karabutaworld gmail.com> writes:
On Friday, 1 January 2021 at 22:07:28 UTC, Selim Ozel wrote:

 I created the simplest possible example as explained by the 
 Vibe-D community in [1]. The exact source code of what I run is 
 in [2].

 On Windows I get a socket handle leak warning on shutdown with 
 crtl+c from terminal after running the executable.


 [...]


 On Ubuntu 20.04 I get leaking drivers warning with the same 
 process.

    [...]


 I really don't know what this is all about but it is at the 
 core of my Vibe-D development. So any pointers you might have 
 would be very helpful to me.

 Thanks in advance.

 S

 [1] 
 https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d
 [2] https://github.com/SelimOzel/vibe_noLeaks


Add this to your dub.json file to fix it

"versions": [ "VibeHighEventPriority" ]


This issue should be fixed in next vibe.d release
Jan 03 2021
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/3/21 6:53 PM, aberba wrote:

 On Friday, 1 January 2021 at 22:07:28 UTC, Selim Ozel wrote:

 I created the simplest possible example as explained by the Vibe-D 
 community in [1]. The exact source code of what I run is in [2].

 On Windows I get a socket handle leak warning on shutdown with crtl+c 
 from terminal after running the executable.


 [...]


 On Ubuntu 20.04 I get leaking drivers warning with the same process.

    [...]


 I really don't know what this is all about but it is at the core of my 
 Vibe-D development. So any pointers you might have would be very 
 helpful to me.

 Thanks in advance.

 S

 [1] https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d
 [2] https://github.com/SelimOzel/vibe_noLeaks


 
 Add this to your dub.json file to fix it
 
 "versions": [ "VibeHighEventPriority" ]
 
 
 This issue should be fixed in next vibe.d release


No, this is a different issue. That issue is when the server doesn't 
shut down so you can't re-bind to the port without killing it with a 
SIGKILL signal.

And thankfully it should be fixed in the latest release (not vibe.d, but 
eventcore I think).

-Steve
Jan 04 2021
prev sibling parent Selim Ozel <sozel wpi.edu> writes:
On Sunday, 3 January 2021 at 23:53:54 UTC, aberba wrote:

 On Friday, 1 January 2021 at 22:07:28 UTC, Selim Ozel wrote:

 I created the simplest possible example as explained by the 
 Vibe-D community in [1]. The exact source code of what I run 
 is in [2].

 On Windows I get a socket handle leak warning on shutdown with 
 crtl+c from terminal after running the executable.


 [...]


 On Ubuntu 20.04 I get leaking drivers warning with the same 
 process.

    [...]


 I really don't know what this is all about but it is at the 
 core of my Vibe-D development. So any pointers you might have 
 would be very helpful to me.

 Thanks in advance.

 S

 [1] 
 https://vibed.org/blog/posts/a-scalable-chat-room-service-in-d
 [2] https://github.com/SelimOzel/vibe_noLeaks


 Add this to your dub.json file to fix it

 "versions": [ "VibeHighEventPriority" ]


 This issue should be fixed in next vibe.d release


Thanks. Not sure if relevant to this one but I came across that 
one as well before [1]. The symptom was "The simple hello world 
app I build with vibe-d does not seem to work on the second 
compile+execution." on an Ubuntu 20.04 EC2.

[1] 
https://forum.dlang.org/post/mailman.6758.1605999004.31109.digitalmars-d puremagic.com
Jan 05 2021
prev sibling parent aberba <karabutaworld gmail.com> writes:
On Friday, 1 January 2021 at 22:07:28 UTC, Selim Ozel wrote:


 [2] https://github.com/SelimOzel/vibe_noLeaks


I don't see anything abnormal in this code though. Will trying it 
later today myself.

Could you show the actual socket code causing that leak?
Jan 08 2021