www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Bug in phobos Thread module?

reply Babele Dunnit <babele.dunnit gmail.com> writes:
Hi all,

I am a newbie with D but experienced C++ developer.. this is my first post, so
I MUST say that D ROCKS! Really impressive language. Walter, you are definitely
the Lord of Compilers. Back to the subject, I am writing a massively
multithreaded piece of code, and after some time I get a "failed to start"
error. I digged in the forums and found someone else with same problem, but no
answer. So was time to dig into Thread sources...

...and I see a static destructor cleaning a single handle (the main thread, I
suppose), but no CloseHandle on any other handle created via _beginthreadex (I
am talking about Windows, I should have specified before, sorry). 

I believe there should be an explicit Thread destructor able to free the handle
via CloseHandle; also, because thread handles under Windows are a limited
resource, probably a RIAA scheme (or explicit "delete" call) should be used,
instead of waiting for the GC to pass by..

So, I added a CloseHandle(mythread.hdl) call and now my handles count (in the
Task Manager) is much more under control...

ciao
Bab
May 11 2007
next sibling parent reply Regan Heath <regan netmail.co.nz> writes:
I suspect you have found a bug, in which case it should be reported in the bug
tracker...  Now, as I've been away for 6 months and forgotten, can someone tell
us both how that's done.

Regan
May 11 2007
next sibling parent david <ta-nospam-zz gmx.at> writes:
Regan Heath schrieb:
 I suspect you have found a bug, in which case it should be reported in the bug
tracker...  Now, as I've been away for 6 months and forgotten, can someone tell
us both how that's done.
 
 Regan

look here: http://d.puremagic.com/issues/index.cgi david
May 11 2007
prev sibling parent "Chris Miller" <chris dprogramming.com> writes:
On Fri, 11 May 2007 18:47:04 -0400, Regan Heath <regan netmail.co.nz>  =

wrote:

 I suspect you have found a bug, in which case it should be reported in=

 the bug tracker...  Now, as I've been away for 6 months and forgotten,=

 can someone tell us both how that's done.

 Regan

It has been reported and confirmed several times: http://d.puremagic.com/issues/show_bug.cgi?id=3D318
May 11 2007
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Babele Dunnit wrote:

 I am a newbie with D but experienced C++ developer.. this is my first post, so
I MUST say that D ROCKS! Really impressive language. Walter, you are definitely
the Lord of Compilers. Back to the subject, I am writing a massively
multithreaded piece of code, and after some time I get a "failed to start"
error. I digged in the forums and found someone else with same problem, but no
answer. So was time to dig into Thread sources...
 
 ...and I see a static destructor cleaning a single handle (the main thread, I
suppose), but no CloseHandle on any other handle created via _beginthreadex (I
am talking about Windows, I should have specified before, sorry). 
 
 I believe there should be an explicit Thread destructor able to free the
handle via CloseHandle; also, because thread handles under Windows are a
limited resource, probably a RIAA scheme (or explicit "delete" call) should be
used, instead of waiting for the GC to pass by..
 
 So, I added a CloseHandle(mythread.hdl) call and now my handles count (in the
Task Manager) is much more under control...

I'm not too experienced with threading, so if you could post a patch that would be welcome.
May 11 2007
next sibling parent reply kenny <funisher gmail.com> writes:
Walter Bright wrote:
 Babele Dunnit wrote:
 
 I am a newbie with D but experienced C++ developer.. this is my first
 post, so I MUST say that D ROCKS! Really impressive language. Walter,
 you are definitely the Lord of Compilers. Back to the subject, I am
 writing a massively multithreaded piece of code, and after some time I
 get a "failed to start" error. I digged in the forums and found
 someone else with same problem, but no answer. So was time to dig into
 Thread sources...

 ...and I see a static destructor cleaning a single handle (the main
 thread, I suppose), but no CloseHandle on any other handle created via
 _beginthreadex (I am talking about Windows, I should have specified
 before, sorry).
 I believe there should be an explicit Thread destructor able to free
 the handle via CloseHandle; also, because thread handles under Windows
 are a limited resource, probably a RIAA scheme (or explicit "delete"
 call) should be used, instead of waiting for the GC to pass by..

 So, I added a CloseHandle(mythread.hdl) call and now my handles count
 (in the Task Manager) is much more under control...

I'm not too experienced with threading, so if you could post a patch that would be welcome.

I also experience the same issue in linux. It happens after I create/delete over 8000 threads. I dunno why 8000 is the magic number, but it is for my machine. I'll try and look into it as well on linux. Just wanted to let you know that the problem also occurs in linux
May 12 2007
parent reply Regan Heath <regan netmail.co.nz> writes:
kenny Wrote:
 I also experience the same issue in linux. It happens after I create/delete
over 8000 threads. I dunno why 8000 is the magic number, but it is for my
machine. I'll try and look into it as well on linux. Just wanted to let you
know that the problem also occurs in linux

Each operating system has a maximum number of handles, it differs for operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath
May 12 2007
parent reply kenny <funisher gmail.com> writes:
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Regan Heath wrote:
 kenny Wrote:
 I also experience the same issue in linux. It happens after I create/delete
over 8000 threads. I dunno why 8000 is the magic number, but it is for my
machine. I'll try and look into it as well on linux. Just wanted to let you
know that the problem also occurs in linux

Each operating system has a maximum number of handles, it differs for operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath

wow, now it's only giving me 382 threads... here is test code. Later today, I will dip into phobos and try and find the problem.
May 12 2007
parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
kenny wrote:

 Regan Heath wrote:
 kenny Wrote:
 I also experience the same issue in linux. It happens after I
 create/delete over 8000 threads. I dunno why 8000 is the magic number,
 but it is for my machine. I'll try and look into it as well on linux.
 Just wanted to let you know that the problem also occurs in linux

Each operating system has a maximum number of handles, it differs for operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath

wow, now it's only giving me 382 threads... here is test code. Later today, I will dip into phobos and try and find the problem.

For what it's worth, the Tango version of Thread has several fixes and changes compared to the Phobos version, and also provides other features like thread local storage and fibers/co-routines. -- Lars Ivar Igesund blog at http://larsivi.net DSource, #d.tango & #D: larsivi Dancing the Tango
May 12 2007
parent reply kenny <funisher gmail.com> writes:
Lars Ivar Igesund wrote:
 kenny wrote:
 
 Regan Heath wrote:
 kenny Wrote:
 I also experience the same issue in linux. It happens after I
 create/delete over 8000 threads. I dunno why 8000 is the magic number,
 but it is for my machine. I'll try and look into it as well on linux.
 Just wanted to let you know that the problem also occurs in linux

operating system, each version of each operating system, and can be defined for each process upon execution as well. Or so I recall from my travels. Regan Heath

today, I will dip into phobos and try and find the problem.

For what it's worth, the Tango version of Thread has several fixes and changes compared to the Phobos version, and also provides other features like thread local storage and fibers/co-routines.

well, it looks like I'll be doing one of two things this weekend :) 1. read http://yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html 2. install tango / tangbos because I have to make a program by wednesday that needs to create unlimited threads over unlimited time (eg a daemon of sorts). I'll check out tango, as it sounds a lot easier than bugfixing tango. Thanks for the tip :) Kenny
May 12 2007
parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
kenny wrote:

 well, it looks like I'll be doing one of two things this weekend :)
 
 1. read http://yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html

I can also suggest http://www.dsource.org/projects/tango/wiki/ChapterThreading
 ... as it sounds a lot easier than bugfixing tango. 

I hope you meant Phobos ;) -- Lars Ivar Igesund blog at http://larsivi.net DSource, #d.tango & #D: larsivi Dancing the Tango
May 12 2007
parent kenny <funisher gmail.com> writes:
Lars Ivar Igesund wrote:
 kenny wrote:
 
 well, it looks like I'll be doing one of two things this weekend :)

 1. read http://yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html

I can also suggest http://www.dsource.org/projects/tango/wiki/ChapterThreading
 ... as it sounds a lot easier than bugfixing tango. 

I hope you meant Phobos ;)

awesome, thanks for the link. Yep, I meant phobos :)
May 12 2007
prev sibling parent Regan Heath <regan netmail.co.nz> writes:
Walter Bright Wrote:
 Babele Dunnit wrote:
 
 I am a newbie with D but experienced C++ developer.. this is my first post, so
I MUST say that D ROCKS! Really impressive language. Walter, you are definitely
the Lord of Compilers. Back to the subject, I am writing a massively
multithreaded piece of code, and after some time I get a "failed to start"
error. I digged in the forums and found someone else with same problem, but no
answer. So was time to dig into Thread sources...
 
 ...and I see a static destructor cleaning a single handle (the main thread, I
suppose), but no CloseHandle on any other handle created via _beginthreadex (I
am talking about Windows, I should have specified before, sorry). 
 
 I believe there should be an explicit Thread destructor able to free the
handle via CloseHandle; also, because thread handles under Windows are a
limited resource, probably a RIAA scheme (or explicit "delete" call) should be
used, instead of waiting for the GC to pass by..
 
 So, I added a CloseHandle(mythread.hdl) call and now my handles count (in the
Task Manager) is much more under control...

I'm not too experienced with threading, so if you could post a patch that would be welcome.

I've no idea how to do a patch as such but I believe all you need is a call to CloseHandle in: extern (Windows) static uint threadstart(void *p) as this wrapper is used for all threads and the stuff at the end of the wrapper removes the thread from allThreads and does cleanup. eg debug (thread) printf("Ending thread %d\n", t.idx); version (Win32) { CloseHandle(t.hdl); t.hdl = cast(thread_hdl)0; } t.state = TS.TERMINATED; allThreads[t.idx] = null; I cannot recall (nor do I have old code I can refer to, sadly) whether there is a similar call required on Linux etc, I suspect not given that pthread_create returns a thread id, as opposed to a handle per-se. I have vague recollections of calling kill, but that may be to forcibly terminate a thread as opposed to cleaning up a handle. Regan Heath
May 12 2007