www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Weird std.stdio threading bug?

reply dsimcha <dsimcha yahoo.com> writes:
The following small test program seems to have a weird deadlock or something:
 It should keep printing the phrase "Doing stuff." forever, but it only gets
through maybe two iterations before its CPU usage does to zero and it stops
printing, at least on my computer.  Has anyone noticed any bad behavior with
std.stdio and multithreading?

import core.thread, std.stdio;

void main() {
    Thread[] myThreads;
    foreach(i; 0..4) {
        myThreads ~= new Thread( { doStuff(); });
        myThreads[$ - 1].start;
    }
}



void doStuff() {
    while(true) {
        synchronized {
            writeln("Doing stuff.");
        }
    }
}


If the writeln line is commented out, this thing keeps executing the empty
loop with measurable CPU usage.
Apr 27 2009
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 27 Apr 2009 18:53:04 -0400, dsimcha <dsimcha yahoo.com> wrote:

 The following small test program seems to have a weird deadlock or  
 something:
  It should keep printing the phrase "Doing stuff." forever, but it only  
 gets
 through maybe two iterations before its CPU usage does to zero and it  
 stops
 printing, at least on my computer.  Has anyone noticed any bad behavior  
 with
 std.stdio and multithreading?

 import core.thread, std.stdio;

 void main() {
     Thread[] myThreads;
     foreach(i; 0..4) {
         myThreads ~= new Thread( { doStuff(); });
         myThreads[$ - 1].start;
     }
 }



 void doStuff() {
     while(true) {
         synchronized {
             writeln("Doing stuff.");
         }
     }
 }


 If the writeln line is commented out, this thing keeps executing the  
 empty
 loop with measurable CPU usage.

Shouldn't you be waiting for the threads to exit at the end of main? I wonder if the GC has been shut down by main exiting. -Steve
Apr 28 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Mon, 27 Apr 2009 18:53:04 -0400, dsimcha <dsimcha yahoo.com> wrote:
 The following small test program seems to have a weird deadlock or
 something:
  It should keep printing the phrase "Doing stuff." forever, but it only
 gets
 through maybe two iterations before its CPU usage does to zero and it
 stops
 printing, at least on my computer.  Has anyone noticed any bad behavior
 with
 std.stdio and multithreading?

 import core.thread, std.stdio;

 void main() {
     Thread[] myThreads;
     foreach(i; 0..4) {
         myThreads ~= new Thread( { doStuff(); });
         myThreads[$ - 1].start;
     }
 }



 void doStuff() {
     while(true) {
         synchronized {
             writeln("Doing stuff.");
         }
     }
 }


 If the writeln line is commented out, this thing keeps executing the
 empty
 loop with measurable CPU usage.

wonder if the GC has been shut down by main exiting. -Steve

I guess you're right. (This test program was written based on a larger program where this effect shows up, and forgetting that was just an oversight.) On the other hand, it still doesn't fix the problem. Also note, in case it's relevant, my OS is win32. import core.thread, std.stdio; void main() { Thread[] myThreads; foreach(i; 0..4) { myThreads ~= new Thread( { doStuff(); }); myThreads[$ - 1].start; } doStuff(); } void doStuff() { while(true) { synchronized { writeln("Doing stuff."); } } }
Apr 28 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 Have you tried synchronizing on an actual object?  I remember some time
 back how Walter proposed removing synchronized as you have written it.
 Not sure what happened for that.
 The way you have written the code, assuming that the synchronized
 statement is doing what you think it's doing, the call to writeln should
 be completely syncrhonous, so multithreading issues or not, it should work.
 -Steve

Good idea, still doesn't work. All of the loops just die after a few iterations, leaving me at 0% CPU usage. This happens on multiple win32 boxes. Could someone please test this on some other OS? import core.thread, std.stdio; class Lock {} Lock lock; void main() { lock = new Lock; Thread[] myThreads; foreach(i; 0..4) { myThreads ~= new Thread( { doStuff(); }); myThreads[$ - 1].start; } doStuff(); } void doStuff() { while(true) { synchronized(lock) { writeln("Doing stuff."); } } }
Apr 28 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Tue, 28 Apr 2009 09:51:54 -0400, dsimcha <dsimcha yahoo.com> wrote:
 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 Have you tried synchronizing on an actual object?  I remember some time
 back how Walter proposed removing synchronized as you have written it.
 Not sure what happened for that.
 The way you have written the code, assuming that the synchronized
 statement is doing what you think it's doing, the call to writeln should
 be completely syncrhonous, so multithreading issues or not, it should
 work.
 -Steve

Good idea, still doesn't work. All of the loops just die after a few iterations, leaving me at 0% CPU usage. This happens on multiple win32 boxes. Could someone please test this on some other OS?

-Steve

2.029. Thanks for your help. I'm just trying to make sure this is a legit bug and maybe understand its underpinnings a little better before I file a bug report, because if it is a legit bug, it's a pretty serious one.
Apr 28 2009
parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Tue, 28 Apr 2009 10:16:21 -0400, dsimcha <dsimcha yahoo.com> wrote:
 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Tue, 28 Apr 2009 09:51:54 -0400, dsimcha <dsimcha yahoo.com> wrote:
 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 Have you tried synchronizing on an actual object?  I remember some


 back how Walter proposed removing synchronized as you have written


 Not sure what happened for that.
 The way you have written the code, assuming that the synchronized
 statement is doing what you think it's doing, the call to writeln


 be completely syncrhonous, so multithreading issues or not, it should
 work.
 -Steve

Good idea, still doesn't work. All of the loops just die after a few iterations, leaving me at 0% CPU usage. This happens on multiple win32 boxes. Could someone please test this on some other OS?

-Steve

2.029. Thanks for your help. I'm just trying to make sure this is a legit bug and maybe understand its underpinnings a little better before I file a bug report, because if it is a legit bug, it's a pretty serious one.

I also tried several different things: Things that worked (continually output "Doing stuff."): 1. removing the synchronized(lock) statement 2. Changing the number of threads to 0 3. Changing the print line to use printf("Doing stuff.\n"); Things that still failed: 4. Changing the number of threads to 1 (which would mean 2 threads, the main thread and the sub thread). The fact that 2 worked and 3 worked indicates to me that it's not simply a bug involving the lock mechanism, it's definitely something to do with writeln. I'd file a bug with the latest code you posted. I tried the original code once on Linux. I get much more printouts than 2 or 3, but it still hangs. So it's cross-platform. -Steve

Done. http://d.puremagic.com/issues/show_bug.cgi?id=2907 Thanks for your help.
Apr 28 2009
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 28 Apr 2009 09:14:15 -0400, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 Shouldn't you be waiting for the threads to exit at the end of main?  I
 wonder if the GC has been shut down by main exiting.
 -Steve

I guess you're right. (This test program was written based on a larger program where this effect shows up, and forgetting that was just an oversight.) On the other hand, it still doesn't fix the problem. Also note, in case it's relevant, my OS is win32. import core.thread, std.stdio; void main() { Thread[] myThreads; foreach(i; 0..4) { myThreads ~= new Thread( { doStuff(); }); myThreads[$ - 1].start; } doStuff(); } void doStuff() { while(true) { synchronized { writeln("Doing stuff."); } } }

Have you tried synchronizing on an actual object? I remember some time back how Walter proposed removing synchronized as you have written it. Not sure what happened for that. The way you have written the code, assuming that the synchronized statement is doing what you think it's doing, the call to writeln should be completely syncrhonous, so multithreading issues or not, it should work. -Steve
Apr 28 2009
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 28 Apr 2009 09:51:54 -0400, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 Have you tried synchronizing on an actual object?  I remember some time
 back how Walter proposed removing synchronized as you have written it.
 Not sure what happened for that.
 The way you have written the code, assuming that the synchronized
 statement is doing what you think it's doing, the call to writeln should
 be completely syncrhonous, so multithreading issues or not, it should  
 work.
 -Steve

Good idea, still doesn't work. All of the loops just die after a few iterations, leaving me at 0% CPU usage. This happens on multiple win32 boxes. Could someone please test this on some other OS?

Latest compiler? D1 D2? -Steve
Apr 28 2009
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 28 Apr 2009 09:59:20 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Tue, 28 Apr 2009 09:51:54 -0400, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 Have you tried synchronizing on an actual object?  I remember some time
 back how Walter proposed removing synchronized as you have written it.
 Not sure what happened for that.
 The way you have written the code, assuming that the synchronized
 statement is doing what you think it's doing, the call to writeln  
 should
 be completely syncrhonous, so multithreading issues or not, it should  
 work.
 -Steve

Good idea, still doesn't work. All of the loops just die after a few iterations, leaving me at 0% CPU usage. This happens on multiple win32 boxes. Could someone please test this on some other OS?

Latest compiler? D1 D2?

import core.Thread Duh. Must be D2 :) Probably latest. -steve
Apr 28 2009
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 28 Apr 2009 10:16:21 -0400, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 On Tue, 28 Apr 2009 09:51:54 -0400, dsimcha <dsimcha yahoo.com> wrote:
 == Quote from Steven Schveighoffer (schveiguy yahoo.com)'s article
 Have you tried synchronizing on an actual object?  I remember some  


 back how Walter proposed removing synchronized as you have written  


 Not sure what happened for that.
 The way you have written the code, assuming that the synchronized
 statement is doing what you think it's doing, the call to writeln  


 be completely syncrhonous, so multithreading issues or not, it should
 work.
 -Steve

Good idea, still doesn't work. All of the loops just die after a few iterations, leaving me at 0% CPU usage. This happens on multiple win32 boxes. Could someone please test this on some other OS?

-Steve

2.029. Thanks for your help. I'm just trying to make sure this is a legit bug and maybe understand its underpinnings a little better before I file a bug report, because if it is a legit bug, it's a pretty serious one.

Yep, I confirm (on WinXP). I get 2 or three outputs and then hang. I also tried several different things: Things that worked (continually output "Doing stuff."): 1. removing the synchronized(lock) statement 2. Changing the number of threads to 0 3. Changing the print line to use printf("Doing stuff.\n"); Things that still failed: 4. Changing the number of threads to 1 (which would mean 2 threads, the main thread and the sub thread). The fact that 2 worked and 3 worked indicates to me that it's not simply a bug involving the lock mechanism, it's definitely something to do with writeln. I'd file a bug with the latest code you posted. I tried the original code once on Linux. I get much more printouts than 2 or 3, but it still hangs. So it's cross-platform. -Steve
Apr 28 2009
prev sibling parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
I posted a bug report for this a few days ago (2890). I got as far as 
finding out that it is a file locking problem caused by what looks like a 
compiler bug re calling the destructor of a struct. The following patch 
to std.stdio works around the problem, but is hardly a fix.

$ diff dmd/src/phobos/std/stdio.d stdio.d
922c922
<         //return LockingTextWriter(this);
---
         return LockingTextWriter(this);

< auto result = LockingTextWriter(this); < return result; ---
         //auto result = LockingTextWriter(this);
         //return result;

On Mon, 27 Apr 2009 22:53:04 +0000, dsimcha wrote:
 The following small test program seems to have a weird deadlock or
 something:
  It should keep printing the phrase "Doing stuff." forever, but it only
  gets
 through maybe two iterations before its CPU usage does to zero and it
 stops printing, at least on my computer.  Has anyone noticed any bad
 behavior with std.stdio and multithreading?
 
 import core.thread, std.stdio;
 
 void main() {
     Thread[] myThreads;
     foreach(i; 0..4) {
         myThreads ~= new Thread( { doStuff(); }); myThreads[$ -
         1].start;
     }
 }
 
 
 
 void doStuff() {
     while(true) {
         synchronized {
             writeln("Doing stuff.");
         }
     }
 }
 
 
 If the writeln line is commented out, this thing keeps executing the
 empty loop with measurable CPU usage.

Apr 30 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Graham St Jack wrote:
 I posted a bug report for this a few days ago (2890). I got as far as 
 finding out that it is a file locking problem caused by what looks like a 
 compiler bug re calling the destructor of a struct. The following patch 
 to std.stdio works around the problem, but is hardly a fix.
 
 $ diff dmd/src/phobos/std/stdio.d stdio.d
 922c922
 <         //return LockingTextWriter(this);
 ---
         return LockingTextWriter(this);

< auto result = LockingTextWriter(this); < return result; ---
         //auto result = LockingTextWriter(this);
         //return result;


Thanks! I checked in your fix. As you saw, std.stdio is full of commented-out code that I wrote in desperation. The copy construction bugs are killing me. Andrei
Apr 30 2009