www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Bug? taskPool.map() with bufSize and writeln() gets stuck

reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
Sorry for the double-post; I have asked the same question on D.learn 
earlier but I think this is more of a question to this forum.

Tested on Ubuntu 11.10 64-bit dmd.

The following program gets stuck during the writeln() call.

- Note that the foo() call alone works fine.

- Also note that the program works fine when there is no writeln() call 
nor foo() call. All elements get processed in that case and the results 
are ignored.

Am I using taskPool.map incorrectly or is this a bug? Can you help 
identify where the problem may be? How is writeln() using the range 
differently than foo() to cause this behavior?

import std.stdio;
import std.parallelism;
import core.thread;

int func(int i)
{
     writeln("processing ", i);
     return i;
}

void main()
{
     auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

     writeln(results);  // <-- Gets stuck HERE

     foo(results); // this works fine
}

void foo(R)(R range)
{
     for ( ; !range.empty; range.popFront()) {
         writeln(range.front);
     }
}

Thank you,
Ali
Feb 10 2012
next sibling parent "MattCodr" <matheus_nab hotmail.com> writes:
On Saturday, 11 February 2012 at 01:31:29 UTC, Ali Çehreli wrote:
 Sorry for the double-post; I have asked the same question on 
 D.learn earlier but I think this is more of a question to this 
 forum.

 Tested on Ubuntu 11.10 64-bit dmd.

 The following program gets stuck during the writeln() call.

 - Note that the foo() call alone works fine.

 - Also note that the program works fine when there is no 
 writeln() call nor foo() call. All elements get processed in 
 that case and the results are ignored.

 Am I using taskPool.map incorrectly or is this a bug? Can you 
 help identify where the problem may be? How is writeln() using 
 the range differently than foo() to cause this behavior?

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
    writeln("processing ", i);
    return i;
 }

 void main()
 {
    auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

    writeln(results);  // <-- Gets stuck HERE

    foo(results); // this works fine
 }

 void foo(R)(R range)
 {
    for ( ; !range.empty; range.popFront()) {
        writeln(range.front);
    }
 }

 Thank you,
 Ali

For what I see, maybe that's because it's a "non-random access ranges". So to access "results", you need access from it's aggregate... try this inside your main: foreach(int r; results) writeln(r); // foo(results); // this works fin
Feb 10 2012
prev sibling next sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Sat, 11 Feb 2012 02:31:29 +0100, Ali =C3=87ehreli <acehreli yahoo.com=
 wrote:

 Sorry for the double-post; I have asked the same question on D.learn  =

 earlier but I think this is more of a question to this forum.

 Tested on Ubuntu 11.10 64-bit dmd.

 The following program gets stuck during the writeln() call.

 - Note that the foo() call alone works fine.

 - Also note that the program works fine when there is no writeln() cal=

 nor foo() call. All elements get processed in that case and the result=

 are ignored.

 Am I using taskPool.map incorrectly or is this a bug? Can you help  =

 identify where the problem may be? How is writeln() using the range  =

 differently than foo() to cause this behavior?

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
      writeln("processing ", i);
      return i;
 }

 void main()
 {
      auto results =3D taskPool.map!func([1,2,3,4,5,6,7,8], 2);

      writeln(results);  // <-- Gets stuck HERE

      foo(results); // this works fine
 }

 void foo(R)(R range)
 {
      for ( ; !range.empty; range.popFront()) {
          writeln(range.front);
      }
 }

 Thank you,
 Ali

Yeah, you have a deadlock in there, it's somewhat hidden though. The issue is that writeln will take lock on stdout once. This will deadl= ock with the lazy processing of the map range.
Feb 11 2012
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 02/11/2012 12:56 AM, Martin Nowak wrote:
 On Sat, 11 Feb 2012 02:31:29 +0100, Ali Çehreli <acehreli yahoo.com> 

 Sorry for the double-post; I have asked the same question on D.learn
 earlier but I think this is more of a question to this forum.

 Tested on Ubuntu 11.10 64-bit dmd.

 The following program gets stuck during the writeln() call.

 - Note that the foo() call alone works fine.

 - Also note that the program works fine when there is no writeln()
 call nor foo() call. All elements get processed in that case and the
 results are ignored.

 Am I using taskPool.map incorrectly or is this a bug? Can you help
 identify where the problem may be? How is writeln() using the range
 differently than foo() to cause this behavior?

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
 writeln("processing ", i);
 return i;
 }

 void main()
 {
 auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2);

 writeln(results); // <-- Gets stuck HERE

 foo(results); // this works fine
 }

 void foo(R)(R range)
 {
 for ( ; !range.empty; range.popFront()) {
 writeln(range.front);
 }
 }

 Thank you,
 Ali

Yeah, you have a deadlock in there, it's somewhat hidden though. The issue is that writeln will take lock on stdout once. This will

 with the lazy processing of the map range.

Thank you. I was trying to visualize the semi-lazy nature of taskPool.map. Now I get what I want when the writeln() call in main() is changed to be on stderr: import std.stdio; import std.parallelism; import core.thread; int func(int i) { writeln("processing ", i); Thread.sleep(dur!"seconds"(1)); return i; } void main() { auto results = taskPool.map!func([1,2,3,4,5,6,7,8], 2); stderr.writeln(results); // <-- now on stderr } Now the output hints at how taskPool.map is semi-lazy: processing 1 processing 2 [processing 3 1, 2processing 4 , 3processing 5 , 4processing 6 , 5, 6processing 7 processing 8 , 7, 8] Good. :) Ali
Feb 11 2012
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Sat, 11 Feb 2012 17:18:21 +0100, Ali =C3=87ehreli <acehreli yahoo.com=
 wrote:

 On 02/11/2012 12:56 AM, Martin Nowak wrote:
  > On Sat, 11 Feb 2012 02:31:29 +0100, Ali =C3=87ehreli <acehreli yaho=

 wrote:
  >
  >> Sorry for the double-post; I have asked the same question on D.lea=

  >> earlier but I think this is more of a question to this forum.
  >>
  >> Tested on Ubuntu 11.10 64-bit dmd.
  >>
  >> The following program gets stuck during the writeln() call.
  >>
  >> - Note that the foo() call alone works fine.
  >>
  >> - Also note that the program works fine when there is no writeln()=

  >> call nor foo() call. All elements get processed in that case and t=

  >> results are ignored.
  >>
  >> Am I using taskPool.map incorrectly or is this a bug? Can you help=

  >> identify where the problem may be? How is writeln() using the rang=

  >> differently than foo() to cause this behavior?
  >>
  >> import std.stdio;
  >> import std.parallelism;
  >> import core.thread;
  >>
  >> int func(int i)
  >> {
  >> writeln("processing ", i);
  >> return i;
  >> }
  >>
  >> void main()
  >> {
  >> auto results =3D taskPool.map!func([1,2,3,4,5,6,7,8], 2);
  >>
  >> writeln(results); // <-- Gets stuck HERE
  >>
  >> foo(results); // this works fine
  >> }
  >>
  >> void foo(R)(R range)
  >> {
  >> for ( ; !range.empty; range.popFront()) {
  >> writeln(range.front);
  >> }
  >> }
  >>
  >> Thank you,
  >> Ali
  >
  > Yeah, you have a deadlock in there, it's somewhat hidden though.
  > The issue is that writeln will take lock on stdout once. This will =

 deadlock
  > with the lazy processing of the map range.

 Thank you.

 I was trying to visualize the semi-lazy nature of taskPool.map. Now I =

 get what I want when the writeln() call in main() is changed to be on =

 stderr:

 import std.stdio;
 import std.parallelism;
 import core.thread;

 int func(int i)
 {
      writeln("processing ", i);
      Thread.sleep(dur!"seconds"(1));
      return i;
 }

 void main()
 {
      auto results =3D taskPool.map!func([1,2,3,4,5,6,7,8], 2);

      stderr.writeln(results);    // <-- now on stderr
 }

 Now the output hints at how taskPool.map is semi-lazy:

 processing 1
 processing 2
 [processing 3
 1, 2processing 4
 , 3processing 5
 , 4processing 6
 , 5, 6processing 7
 processing 8
 , 7, 8]

 Good. :)

 Ali

right output.
Feb 11 2012