www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 4307] New: spawn()'ed thread doesn't terminate

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307

           Summary: spawn()'ed thread doesn't terminate
           Product: D
           Version: D2
          Platform: x86
        OS/Version: Windows
            Status: NEW
          Severity: normal
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: torhu yahoo.com



Using DMD 2.047.

This example hangs after printing '9'.  From reading Andrei's book, my
understanding is that the spawned thread should terminate automatically when
its owner thread terminates.  But that doesn't happen here.

---
import std.concurrency;
import std.stdio;

void f()
{
    for (;;)
    {
        int i = receiveOnly!int();
        writeln(i);
    }
}

void main()
{
    Tid tid = spawn(&f);
    foreach (int i; 0..10)
    {
        send(tid, i);
    }
}
---

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jun 13 2010
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
The main thread is somewhat special in D--it doesn't actually terminate until
all other threads have terminated.  And because the "OwnerTerminated" message
is sent in a static dtor, the ordering is wrong for having spawned threads
receive this message.  I'll leave it up to Andrei to decide whether this is
correct behavior or if the OwnerTerminated message should be sent when D main()
exits (I'm leaning towards this latter behavior myself).  If a change is
necessary I'll have to add a stack for onMainExit callbacks to be executed in
druntime.
Jun 15 2010
parent Sean Kelly <sean invisibleduck.org> writes:
Never mind, I'll fix it.  This will require adding an atexit() sort of routine
to druntime, which I believe was requested in another ticket anyway.
Jun 16 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307


Jonathan M Davis <jmdavisProg gmx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jmdavisProg gmx.com



PDT ---
I don't know if this is the same problem or not, but the most threads that I
seem to be able to create in a program is 505 or 506.

import std.concurrency;
import std.stdio;

void main()
{
    int currThreads = 0;
    enum maxThreads = 6;

    size_t totalThreads = 0;
    auto recProc = (Tid tid)
                   {
                       writeln(++totalThreads);
                   };

    for(size_t i = 0; i < 1_000; ++i)
    {
        if(currThreads < maxThreads)
            receiveTimeout(1, recProc);
        else
            receive(recProc);

        spawn(&threadFunc, thisTid);
    }

    while(currThreads > 0)
        receive(recProc);
}

void threadFunc(Tid parentTid)
{
    send(parentTid, thisTid);
}


prints out to either 505 or 506 and then throws an exception:

core.thread.ThreadException: Error creating thread
----------------
./d(void core.thread.Thread.start()) [0x80902b0]
./d(_D3std11concurrency34__T6_spawnTS3std11concurrency3TidZ6_spawnFbPFS3std11concurrency3TidZvS3std11concurrency3TidZS3std11concurrency3Tid+0x7f)
[0x808c3c7]
./d(_D3std11concurrency33__T5spawnTS3std11concurrency3TidZ5spawnFPFS3std11concurrency3TidZvS3std11concurrency3TidZS3std11concurrency3Tid+0x10)
[0x808c344]
./d(_Dmain+0x6b) [0x8087d0f]
./d(extern (C) int rt.dmain2.main(int, char**)) [0x8091a56]
./d(extern (C) int rt.dmain2.main(int, char**)) [0x80919b0]
./d(extern (C) int rt.dmain2.main(int, char**)) [0x8091a9a]
./d(extern (C) int rt.dmain2.main(int, char**)) [0x80919b0]
./d(main+0x96) [0x8091956]
/usr/lib32/libc.so.6(__libc_start_main+0xe6) [0xf756ec76]
./d() [0x8087bf1]


My best guess is that the threads aren't really terminating, so the OS is
running out of threads.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 01 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PDT ---
If I put a print statement at the end of exec() in std.concurrency._spawn(), it
appears to print out just fine every time that a spawned thread is supposed to
be terminating, so whatever the problem is, I think that it pretty much has to
be in core.Thread (or maybe dmd itself, depending on what's causing the issue)
rather than std.concurrency.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 01 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PDT ---
I think what's happening is that pthread_join() never gets called on a thread
started with spawn(), and IIRC, if pthread_join() never gets called on a
thread, it never actually terminates. The only place that I see pthread_join()
getting called is in Thread.join(), and that never gets called for threads
started with spawn(). What we really want here, I think, is for threads which
succesfully terminate to just join to their parent thread themselves without
the parent thread having to call join() on them, but I'm not sure that you can
really do that with pthreads. Assuming that the parent thread has to join() on
all threads created with pthread_create(), we're going to need to find a way to
get the parent thread to call join() on its spawned threads. About all I can
think of is to have a thread whose entire job is to create threads and and make
sure that they join. But it's been too long since I had to deal with pthreads
for me to remember all of the details.

In any case, I believe that the problem stems from the fact that the spawned
threads are never actually joined.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 01 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PDT ---
Actually, what I think needs to happen is for there to be a way to start
threads as detached rather than joinable and have spawn() start detached
threads rather than joinable threads.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 01 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PDT ---
Actually, the program I posted earlier was buggy (that's what I get for
simplifying code and not studying the simplified version enough, I guess).
Here's the corrected version:

import std.concurrency;
import std.stdio;

void main()
{
    int currThreads = 0;
    enum maxThreads = 6;

    size_t totalThreads = 0;
    auto recProc = (Tid tid)
                   {
                       writeln(++totalThreads);
                       --currThreads;
                   };

    for(size_t i = 0; i < 1_000; ++i)
    {
        if(currThreads < maxThreads)
            receiveTimeout(1, recProc);
        else
            receive(recProc);

        spawn(&threadFunc, thisTid);
        ++currThreads;
    }

    while(currThreads > 0)
        receive(recProc);
}

void threadFunc(Tid parentTid)
{
    send(parentTid, thisTid);
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 02 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PDT ---
Created an attachment (id=797)
Patch for core.thread for linux with changes based off of dmd 2.050.

Here's a patch for core.thread which appears to solve the problem on Linux.
Essentially, I made it so that you can give start() a bool to tell it to start
the thread as detached (as in pthread_detach(), not detached from the D
runtime), which I chose to call joinable (to distinguish from being detached
from the D runtime). So, spawn() can call thread.start(false), and then the
thread should terminate properly without needing to be joined. The changes are
essentially Posix only, so if this problem is also on Windows, then other
changes may be required for Windows and/or MacOS X, but I know essentially
nothing about threads on either of those systems. So, I don't know if this
patch is the best overall solution, but it does appear to fix the problem on
Linux, so even if it's not enough of a fix, it should at least help solve the
problem.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 02 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PDT ---
Created an attachment (id=798)
Patch for std.concurrency for linux with changes based off of dmd 2.050.

The patch for std.concurrency to go with the patch for core.thread.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 02 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307


Jonathan M Davis <jmdavisProg gmx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------

           obsolete|                            |



PDT ---
Created an attachment (id=799)
Patch for core.thread for linux with changes based off of dmd 2.050.

Apparently, I made a couple of errors in my patch, so here's update version.
Unfortunately, while these changes seem to fix all of the simple cases that
I've tried, my more complicated programs still fail to terminate properly, so
there appear to be issues beyond spawned threads being joinable instead of
detached.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 02 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PDT ---
Ah, it looks like the problems with my more complex applications have to do
with passing functions with incorrect signatures to receive(), so my patch does
seem to do the trick (on Linux at least).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 02 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307


Jonathan M Davis <jmdavisProg gmx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
          Component|DMD                         |druntime
         OS/Version|Windows                     |All
           Severity|normal                      |major



PST ---
Actually, I'm going to bump this up to major, since spawn is pretty much
useless in my experience without this being fixed. Also, I'm 99.99999999999999%
sure it's a druntime bug, so I'm moving it to druntime (which I probably would
have done before had I noticed that it was marked as dmd).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 24 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307


Sean Kelly <sean invisibleduck.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
                 CC|                            |sean invisibleduck.org



PST ---
I don't think threads can start as detached, because the GC needs to interact
with them.  I'll give this another look though.  I also fixed a major
timing-related bug since 2.051 was released, which may help here as wells.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 25 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PST ---
I don't know exactly what the GC requires with regards to threads, but when I
was talking about starting a thread as detached, I meant detached in the
pthreads sense, not the GC sense, like core.thread generally talks about with
functions like thread_detachThis(). Spawned threads obviously need to be
attached to the GC. The problem is that they can't have to be joined unless
there's somehow a thread somewhere which cleans them up. Spawned threads are
essentially supposed to clean themselves up and go away when they terminate,
and that essentially means that they need to be detached in the pthread sense,
since the programmer isn't going to have the parent thread join it when it's
done.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 25 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PST ---
Fixed in the current revision.  The threads exit with a MessageMismatch
exception though, which seems like a QOI issue.  They should probably exit with
an OwnerTerminated message instead (which is the message that triggered the
mismatch).  I've just changed the behavior so LinkTerminated and
OwnerTerminated messages are still thrown as-is instead of being considered a
MessageMismatch.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 03 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307




PST ---
What about joining the spawned threads? From what I recall, there was no place
that join was called on them, so if I understand correctly, they'll continue to
exist until the program terminates. And I believe that there are a finite
number of threads allowed at one time, so it would be a problem if spawned
threads continued to exist after they're done executing. If spawned threads are
joinable, they need to be joined or they'll never actually terminate (with
pthreads anyway) - unless I'm misunderstanding something.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 03 2011
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=4307


Jonathan M Davis <jmdavisProg gmx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|                            |FIXED



PDT ---
I believe that this has been long since fixed - certainly all of the programs
here seem to work properly. If someone runs into similar problems, please open
a new bug for them.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
May 21 2013