www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Multitasking question: Making a picture viewer, loading lots of images

reply realhet <real_het hotmail.com> writes:
Hi,

I'm writing a picture browser program and I'd like to ask for a 
bit of assistance.

My app is basically a Win32 window with an OpenGL surface. 
Currently I'm able to display a lot of pictures in it. The 
decompression of the bitmaps take place at the beginning of the 
program.

I want to make this more user friendly, so at first I only draw 
placeholders for the pictures while the actual loading and 
decompression is happening on worker threads.

I used std.concurrency.spawn. The parameters are the filename, 
and an index. After the decompression I let the main thread know 
about there is a bitmap it should upload to a texture later. I do 
it using synchronize, it works without error and was really easy 
to implements in D.

My problem is that when I launch 1000 spawn(), the work overload 
on the main thread is so heavy that the 60FPS timer which is 
refreshing the windows never can be called. Obviously spawn() is 
not the right choice for this.

In my previous projects I addressed this by creating 
NumberOfProcessorThreads threads,and locked them on specific 
ProcessorThreads by using ProcessAffinity, so the main thread has 
enough time to generate fluid user interface while the other 
worker threads used 100%CPU for the fastest processing.

I also remember that with OPENMP I was also able to lock workers 
onto specific ProcessorThreads, it also worked fine.

But how can I do this elegantly in D?

Thanks in advance!


(Also I'm not sure how spaw()-n is working exactly. Is it creates 
a new process for every spawn() call, or is it assigns the task 
to one of the N worker processes? 1000 CreateProcess() api calls 
would be not so nice. I just wanna know it in order to avoid it.)
Apr 02
next sibling parent reply DanielG <simpletangent gmail.com> writes:
I'm no threading expert but 1000 - if those indeed are real 
threads (like, preemptive OS threads) - is wayyyy too many.

My understanding is that there's no benefit to creating more than 
the number of hyperthreads your CPU supports (minus your main 
thread?). So you'd want a work queue of all the available 
work/images, and then some reasonable number of threads (4-12 
depending on core count / hyperthreading) taking work as they 
need it from that queue.

You should probably be looking at std.parallelism (TaskPool etc) 
for this. Perhaps somebody can provide a more detailed how-to ...
Apr 02
parent realhet <real_het hotmail.com> writes:
On Wednesday, 3 April 2019 at 04:09:45 UTC, DanielG wrote:
 My understanding is that there's no benefit to creating more 
 than the number of hyperthreads your CPU
Yes. I'm just searching for a higher abstraction to manage this easily in D. I've already did it years ago on Delphi. (-> Manage N worker threads, and only put light stress on the thread which is running on the same core as the main thread.) Later it was easier to make it in OpenMP, and now I'm just looking for an even prettier way in DLang. Also it's good to know what happens inside. I've checked Resource Monitor, and as you said, the thread count reached above 1000 in the first 3 seconds of running the program. But after that something unusual happens: While my overall CPU usage was still high, the number of threads dropped down to 15, and the remaining 20 seconds of processing was don like this. But still, while all the processing, the main thread got no cpu% at all, it was able to render only the very first frame, the one that displayed the empty 'loading in progress' images. After 23 sec I was able to see all the 1000 pictures.
You should probably be looking at std.parallelism (TaskPool etc) 
for this.
Thank you for mentioning, this std.parallelism.Task.executeOnNewThread will be the next test. Is that behaving the same way like 'spawn'? From its name, I don't think so, but I'll find out.
Apr 03
prev sibling parent Aldo <aldocd4 outlook.com> writes:
On Wednesday, 3 April 2019 at 02:59:51 UTC, realhet wrote:
 Hi,

 I'm writing a picture browser program and I'd like to ask for a 
 bit of assistance.

 [...]
Hello, maybe the following links can be useful: http://ddili.org/ders/d.en/parallelism.html http://ddili.org/ders/d.en/fibers.html
Apr 03