www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - spawn X different workers & wait for results from all of them

reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
Hi, I'm not sure how to best implement the following:

1. I have 4 different tasks to do.
2. All can run in parallel
3. Every task will return some result that I need

Now how to best do it? When using receive() it fires on the first hit 
and the function continues. It's like a receive(OR), one hit and it's 
done. So, I would need something like a receive(ALL) that continues 
only of all results (messages) have been received.

Is something like this available or do I have to build it myself?

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Sep 03 2015
next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
On Thu, 03 Sep 2015 18:50:21 +0200, Robert M. Münch wrote:

 Hi, I'm not sure how to best implement the following:
 
 1. I have 4 different tasks to do.
 2. All can run in parallel 3. Every task will return some result that I
 need
 
 Now how to best do it? When using receive() it fires on the first hit
 and the function continues. It's like a receive(OR), one hit and it's
 done. So, I would need something like a receive(ALL) that continues only
 of all results (messages) have been received.
 
 Is something like this available or do I have to build it myself?
How would receive know? If you're using std.concurrency, the receiving function needs to encode doneness, e.g. const numJobs = 4; foreach (_; 0 .. numJobs) receive(...); Or you could use std.parallelism: foreach (pieceOfWork; parallel(listOfWork)) doIt(pieceOfWork);
Sep 04 2015
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2015-09-04 17:32:48 +0000, Justin Whear said:

 How would receive know?
Well, it could be pretty simple. At the moment: receive( int ..., long ..., myStruct ... ) Will wait for one out of the three. So it's an OR. reveive_all( int ..., long ..., myStruct ... ) would finish if every type was returned once (maybe at least once). With this I would have an AND combination.
 If you're using std.concurrency, the receiving
 function needs to encode doneness, e.g.
 
 const numJobs = 4;
 foreach (_; 0 .. numJobs)
 	receive(...);
That's what you can do if the type is the same. Or just put four receives sequentially in the code. The not so nice side effect is, that you define an order for processing but the tasks might finish in an other order. So, possible work is blocked.
 Or you could use std.parallelism:
 
 foreach (pieceOfWork; parallel(listOfWork))
 	doIt(pieceOfWork);
My "pieceOfWork" is not the same. So I don't have the case: Do 4 time this 1thing. Instead, do 1 time these 4 things. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Sep 05 2015
parent Justin Whear <justin economicmodeling.com> writes:
On Sat, 05 Sep 2015 12:21:33 +0200, Robert M. Münch wrote:

 My "pieceOfWork" is not the same. So I don't have the case: Do 4 time
 this 1thing. Instead, do 1 time these 4 things.
Ah, so you want to receive one each of various types? Something like this might work (untested): // These could be inferred using std.traits.ReturnType alias ResultTypes = AliasSeq!(int, float, Baz); bool received(T) = false; bool receivedAll() { foreach (T; ResultTypes) if (!received!T) return false; return true; } while (!receivedAll) { receive( (int x) { received!int = true; /* other work... */ }, (float x) { received!float = true; /* other work... */ }, (Baz x) { received!Baz = true; /* other work... */ } ); }
Sep 08 2015
prev sibling parent reply "thedeemon" <dlang thedeemon.com> writes:
On Thursday, 3 September 2015 at 16:50:21 UTC, Robert M. Münch 
wrote:
 Hi, I'm not sure how to best implement the following:

 1. I have 4 different tasks to do.
 2. All can run in parallel
 3. Every task will return some result that I need

 Now how to best do it?
I think the Task and taskPool from std.parallelism are a good fit. Something like: auto task1 = task!fun1(params1); auto task2 = task!fun2(params2); auto task3 = task!fun3(params3); auto task4 = task!fun4(params4); taskPool.put(task1); taskPool.put(task2); taskPool.put(task3); taskPool.put(task4); auto res1 = task1.workForce(); auto res2 = task2.workForce(); auto res3 = task3.workForce(); auto res4 = task4.workForce(); //here we have all the results, they were calculated in parallel
Sep 05 2015
parent =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2015-09-05 15:44:02 +0000, thedeemon said:

 I think the Task and taskPool from std.parallelism are a good fit.
 Something like:
 
 auto task1 = task!fun1(params1);
 auto task2 = task!fun2(params2);
 auto task3 = task!fun3(params3);
 auto task4 = task!fun4(params4);
 
 taskPool.put(task1);
 taskPool.put(task2);
 taskPool.put(task3);
 taskPool.put(task4);
 
 auto res1 = task1.workForce();
 auto res2 = task2.workForce();
 auto res3 = task3.workForce();
 auto res4 = task4.workForce();
 
 //here we have all the results, they were calculated in parallel
Thanks, that looks pretty good. Going to x-check how this could be used. -- Robert M. Münch http://www.saphirion.com smarter | better | faster
Sep 05 2015