www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Range and poolTask

reply moechofe <truc moechofe.com> writes:
I wonder if it is possible to write something like this:
---
// taskPool.distribute -- take a range and distribute entries to 
different threads.
dirEntries().distribute(function(R1,R2)(R1 from,R2 to){
	from
	.filter!xxx
	.map!yyy
	.tee!zzz(to);
})
.each!www;
---
This would be great.
Jun 06 2016
parent reply Rene Zwanenburg <renezwanenburg gmail.com> writes:
On Monday, 6 June 2016 at 09:32:30 UTC, moechofe wrote:
 I wonder if it is possible to write something like this:
 ---
 // taskPool.distribute -- take a range and distribute entries 
 to different threads.
 dirEntries().distribute(function(R1,R2)(R1 from,R2 to){
 	from
 	.filter!xxx
 	.map!yyy
 	.tee!zzz(to);
 })
 .each!www;
 ---
 This would be great.
This might be useful: http://dlang.org/phobos/std_parallelism.html#.TaskPool Or, more specifically, http://dlang.org/phobos/std_parallelism.html#.TaskPool.amap http://dlang.org/phobos/std_parallelism.html#.TaskPool.map
Jun 06 2016
parent reply moechofe <truc moechofe.com> writes:
On Monday, 6 June 2016 at 09:38:32 UTC, Rene Zwanenburg wrote:
 http://dlang.org/phobos/std_parallelism.html#.TaskPool

 Or, more specifically,

 http://dlang.org/phobos/std_parallelism.html#.TaskPool.amap
 http://dlang.org/phobos/std_parallelism.html#.TaskPool.map
The functions passed to map or amap take the type of the element of the range as argument, but not a range itself.
Jun 06 2016
parent reply Rene Zwanenburg <renezwanenburg gmail.com> writes:
On Monday, 6 June 2016 at 10:26:11 UTC, moechofe wrote:
 The functions passed to map or amap take the type of the 
 element of the range as argument, but not a range itself.
Right. I don't think I understand what the semantics of your example would be though.. Could you elaborate a bit?
Jun 06 2016
parent moechofe <truc moechofe.com> writes:
On Monday, 6 June 2016 at 11:25:00 UTC, Rene Zwanenburg wrote:
 Could you elaborate a bit?
Yes. I have an InputRange and need to pass it throughout a couple of iteration and manipulation functions such as filter, map and finishing by grouping with fold. Like: ---- myrange .filter!xxx .map!yyy .tee!zzz .fold!www ---- This process take a lot of time, so I decide to put it in parallel. I did something like this: ---- Appender!www output; foreach(iii; parallel(myrange)) { only(iii) .filter!xxx .map!yyy .tee!zzz .copy(output) } output.fold!www; ---- only() will create a Range with one item in it. Also, I can't group directly. I need to use a second Range. It works but it's not elegant as the non-working example above is.
Jun 06 2016