www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Bug? taskPool.map() with bufSize and writeln() gets stuck

reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
Sorry for the double-post; I have asked the same question on D.learn 
earlier but I think this is more of a question to this forum.

Tested on Ubuntu 11.10 64-bit dmd.

The following program gets stuck during the writeln() call.

- Note that the foo() call alone works fine.

- Also note that the program works fine when there is no writeln() call 
nor foo() call. All elements get processed in that case and the results 
are ignored.

Am I using taskPool.map incorrectly or is this a bug? Can you help 
identify where the problem may be? How is writeln() using the range 
differently than foo() to cause this behavior?

import std.stdio;
import std.parallelism;
import core.thread;

int func(int i)
{
     writeln("processing ", i);
     return i;
}

void main()
{
     auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

     writeln(results);  // <-- Gets stuck HERE

     foo(results); // this works fine
}

void foo(R)(R range)
{
     for ( ; !range.empty; range.popFront()) {
         writeln(range.front);
     }
}

Thank you,
Ali
Feb 10 2012
next sibling parent "MattCodr" <matheus_nab hotmail.com> writes:
On Saturday, 11 February 2012 at 01:31:29 UTC, Ali Çehreli wrote:
 Sorry for the double-post; I have asked the same question on 
 D.learn earlier but I think this is more of a question to this 
 forum.

 Tested on Ubuntu 11.10 64-bit dmd.

 The following program gets stuck during the writeln() call.

 - Note that the foo() call alone works fine.

 - Also note that the program works fine when there is no 
 writeln() call nor foo() call. All elements get processed in 
 that case and the results are ignored.

 Am I using taskPool.map incorrectly or is this a bug? Can you 
 help identify where the problem may be? How is writeln() using 
 the range differently than foo() to cause this behavior?

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
    writeln("processing ", i);
    return i;
 }

 void main()
 {
    auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

    writeln(results);  // <-- Gets stuck HERE

    foo(results); // this works fine
 }

 void foo(R)(R range)
 {
    for ( ; !range.empty; range.popFront()) {
        writeln(range.front);
    }
 }

 Thank you,
 Ali

For what I see, maybe that's because it's a "non-random access ranges". So to access "results", you need access from it's aggregate... try this inside your main: foreach(int r; results) writeln(r); // foo(results); // this works fin
Feb 10 2012
prev sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Sat, 11 Feb 2012 02:31:29 +0100, Ali =C3=87ehreli <acehreli yahoo.com=
 wrote:

 Sorry for the double-post; I have asked the same question on D.learn  =

 earlier but I think this is more of a question to this forum.

 Tested on Ubuntu 11.10 64-bit dmd.

 The following program gets stuck during the writeln() call.

 - Note that the foo() call alone works fine.

 - Also note that the program works fine when there is no writeln() cal=

l =
 nor foo() call. All elements get processed in that case and the result=

s =
 are ignored.

 Am I using taskPool.map incorrectly or is this a bug? Can you help  =

 identify where the problem may be? How is writeln() using the range  =

 differently than foo() to cause this behavior?

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
      writeln("processing ", i);
      return i;
 }

 void main()
 {
      auto results =3D taskPool.map!func([1,2,3,4,5,6,7,8], 2);

      writeln(results);  // <-- Gets stuck HERE

      foo(results); // this works fine
 }

 void foo(R)(R range)
 {
      for ( ; !range.empty; range.popFront()) {
          writeln(range.front);
      }
 }

 Thank you,
 Ali

Yeah, you have a deadlock in there, it's somewhat hidden though. The issue is that writeln will take lock on stdout once. This will deadl= ock with the lazy processing of the map range.
Feb 11 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 02/11/2012 12:56 AM, Martin Nowak wrote:
 On Sat, 11 Feb 2012 02:31:29 +0100, Ali Çehreli <acehreli yahoo.com> 

wrote:
 Sorry for the double-post; I have asked the same question on D.learn
 earlier but I think this is more of a question to this forum.

 Tested on Ubuntu 11.10 64-bit dmd.

 The following program gets stuck during the writeln() call.

 - Note that the foo() call alone works fine.

 - Also note that the program works fine when there is no writeln()
 call nor foo() call. All elements get processed in that case and the
 results are ignored.

 Am I using taskPool.map incorrectly or is this a bug? Can you help
 identify where the problem may be? How is writeln() using the range
 differently than foo() to cause this behavior?

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
 writeln("processing ", i);
 return i;
 }

 void main()
 {
 auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

 writeln(results); // <-- Gets stuck HERE

 foo(results); // this works fine
 }

 void foo(R)(R range)
 {
 for ( ; !range.empty; range.popFront()) {
 writeln(range.front);
 }
 }

 Thank you,
 Ali

Yeah, you have a deadlock in there, it's somewhat hidden though. The issue is that writeln will take lock on stdout once. This will

deadlock
 with the lazy processing of the map range.

Thank you. I was trying to visualize the semi-lazy nature of taskPool.map. Now I get what I want when the writeln() call in main() is changed to be on stderr: import std.stdio; import std.parallelism; import core.thread; int func(int i) { writeln("processing ", i); Thread.sleep(dur!"seconds"(1)); return i; } void main() { auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2); stderr.writeln(results); // <-- now on stderr } Now the output hints at how taskPool.map is semi-lazy: processing 1 processing 2 [processing 3 1, 2processing 4 , 3processing 5 , 4processing 6 , 5, 6processing 7 processing 8 , 7, 8] Good. :) Ali
Feb 11 2012
parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Sat, 11 Feb 2012 17:18:21 +0100, Ali =C3=87ehreli <acehreli yahoo.com=
 wrote:

 On 02/11/2012 12:56 AM, Martin Nowak wrote:
  > On Sat, 11 Feb 2012 02:31:29 +0100, Ali =C3=87ehreli <acehreli yaho=

o.com> =
 wrote:
  >
  >> Sorry for the double-post; I have asked the same question on D.lea=

rn
  >> earlier but I think this is more of a question to this forum.
  >>
  >> Tested on Ubuntu 11.10 64-bit dmd.
  >>
  >> The following program gets stuck during the writeln() call.
  >>
  >> - Note that the foo() call alone works fine.
  >>
  >> - Also note that the program works fine when there is no writeln()=

  >> call nor foo() call. All elements get processed in that case and t=

he
  >> results are ignored.
  >>
  >> Am I using taskPool.map incorrectly or is this a bug? Can you help=

  >> identify where the problem may be? How is writeln() using the rang=

e
  >> differently than foo() to cause this behavior?
  >>
  >> import std.stdio;
  >> import std.parallelism;
  >> import core.thread;
  >>
  >> int func(int i)
  >> {
  >> writeln("processing ", i);
  >> return i;
  >> }
  >>
  >> void main()
  >> {
  >> auto results =3D taskPool.map!func([1,2,3,4,5,6,7,8], 2);
  >>
  >> writeln(results); // <-- Gets stuck HERE
  >>
  >> foo(results); // this works fine
  >> }
  >>
  >> void foo(R)(R range)
  >> {
  >> for ( ; !range.empty; range.popFront()) {
  >> writeln(range.front);
  >> }
  >> }
  >>
  >> Thank you,
  >> Ali
  >
  > Yeah, you have a deadlock in there, it's somewhat hidden though.
  > The issue is that writeln will take lock on stdout once. This will =

=
 deadlock
  > with the lazy processing of the map range.

 Thank you.

 I was trying to visualize the semi-lazy nature of taskPool.map. Now I =

=
 get what I want when the writeln() call in main() is changed to be on =

=
 stderr:

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
      writeln("processing ", i);
      Thread.sleep(dur!"seconds"(1));
      return i;
 }

 void main()
 {
      auto results =3D taskPool.map!func([1,2,3,4,5,6,7,8], 2);

      stderr.writeln(results);    // <-- now on stderr
 }

 Now the output hints at how taskPool.map is semi-lazy:

 processing 1
 processing 2
 [processing 3
 1, 2processing 4
 , 3processing 5
 , 4processing 6
 , 5, 6processing 7
 processing 8
 , 7, 8]

 Good. :)

 Ali

Going with foreach(e; results) will work too and you can write to the = right output.
Feb 11 2012