www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Parallel foreach gets executed one after another

reply Thorsten Sommer <vektoren gmail.com> writes:
Dear Community.

Thank you for helping me so much in the last thread concerning 
our AI library 
(https://forum.dlang.org/thread/rrpmgzqqtkqgeicjdgps forum.dlang.org). Thanks
to you, we can now debug our AI library with GDB. My suspicion was confirmed:
At least one of our parallel foreach loops will only be processed item by item
instead of parallel.

It is the loop that contains the largest workload: An external 
process is started for each agent. I log the PID, so I see that 
the processes are started one after the other.

Our foreach loops are constructed as:

auto taskPool = new TaskPool();
foreach (n, individual; taskPool.parallel(individuals, 
params.workUnitSize4ParallelExecution)) {
     [...]
     pipes = pipeProcess(args, Redirect.stdout | Redirect.stdin);
     [...]
}

taskPool.finish(true);

The looped individuals are sourced by such a statement:
return this.individuals.filter!( [...] ).array;

Currently, for debugging, params.workUnitSize4ParallelExecution 
is set to 100 and individuals contain approx. 10 agents. 
Therefore, we would expect 10 external processes to be started 
simultaneously. Each process runs approx. 4 seconds. Accordingly, 
the processing should take about 4 seconds instead of 40 seconds.

Do you have any ideas what might be the reason for this? If you 
don't know a solution, then maybe approaches to further 
investigation?


Best regards
Thorsten
Sep 16 2017
parent Thorsten Sommer <vektoren gmail.com> writes:
Okay: Sometimes it is enough to write down the problem so that 
someone outside the team understands it. Obviously, 
workUnitSize4ParallelExecution must be set to 1 for all to run 
parallel. I have read the documentation on this subject just 
today and I have overlooked it anyway.
Sep 16 2017