www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Can D do Hard-RT ?

reply ninjadroid gazuga.net writes:
This is NOT flamebait!

I like D.  I want the answer to the question posed in the subject of this post
to be a hearty and resounding "yes!"  However, I have some concerns that such
won't be the case.

D is a GC'd language through and through, and I'm not interested in trying to
extirpate that quality.  I imagine that if I wrote a program that didn't tickle
the GC in any way, shape, or form (i.e. 100% static memory), then it would be
Hard-RT compliant.  However, it probably wouldn't be terribly useful.

The specific scenario I'm interested in is multi-threaded programming with one
Hard-RT thread, communicating with the non-RT threads via a lock-free
ring-buffer.  The Hard-RT thread would be very well behaved and not do one iota
of memory allocation.  Conversely, the other threads, especially the GUI thread,
would.

The D spec says that all threads must pause when the GC runs, regardless of
which thread it runs in.  This seems to invalidate my design's RT-ness.  The
obvious solution of making the program multiprocess (barf) isn't too useful,
because that would make adjusting the RT thread's memory allocation a rather
convoluted process.  With the current design, I can adjust the RT thread's
memory structures in one of the non-RT threads, and protect the data with a
mutex that is try_locked().  A multi-process solution would be markedly uglier,
and I can't think of any method that would be 100% correct too boot; tickling
the GC seems inevitable.

Is there any way around this?  Is it even possible to write a GC that doesn't
stop up all threads?
Sep 09 2004
next sibling parent "Vathix" <vathixSpamFix dprogramming.com> writes:
<ninjadroid gazuga.net> wrote in message
news:chqq71$k0o$1 digitaldaemon.com...
 This is NOT flamebait!

 I like D.  I want the answer to the question posed in the subject of this

 to be a hearty and resounding "yes!"  However, I have some concerns that

 won't be the case.

 D is a GC'd language through and through, and I'm not interested in trying

 extirpate that quality.  I imagine that if I wrote a program that didn't

 the GC in any way, shape, or form (i.e. 100% static memory), then it would

 Hard-RT compliant.  However, it probably wouldn't be terribly useful.

 The specific scenario I'm interested in is multi-threaded programming with

 Hard-RT thread, communicating with the non-RT threads via a lock-free
 ring-buffer.  The Hard-RT thread would be very well behaved and not do one

 of memory allocation.  Conversely, the other threads, especially the GUI

 would.

 The D spec says that all threads must pause when the GC runs, regardless

 which thread it runs in.  This seems to invalidate my design's RT-ness.

 obvious solution of making the program multiprocess (barf) isn't too

 because that would make adjusting the RT thread's memory allocation a

 convoluted process.  With the current design, I can adjust the RT thread's
 memory structures in one of the non-RT threads, and protect the data with

 mutex that is try_locked().  A multi-process solution would be markedly

 and I can't think of any method that would be 100% correct too boot;

 the GC seems inevitable.

 Is there any way around this?  Is it even possible to write a GC that

 stop up all threads?

You can still use C's malloc() and free(), and create threads that are separate from the GC using C functions, but you can't easily use the GC with them.
Sep 09 2004
prev sibling next sibling parent reply WOB <wb sapo.pt> writes:
On Thu, 9 Sep 2004 23:49:53 +0000 (UTC), ninjadroid gazuga.net wrote:

This is NOT flamebait!

I like D.  I want the answer to the question posed in the subject of this post
to be a hearty and resounding "yes!"  However, I have some concerns that such
won't be the case.

D is a GC'd language through and through, and I'm not interested in trying to
extirpate that quality.  I imagine that if I wrote a program that didn't tickle
the GC in any way, shape, or form (i.e. 100% static memory), then it would be
Hard-RT compliant.  However, it probably wouldn't be terribly useful.

The specific scenario I'm interested in is multi-threaded programming with one
Hard-RT thread, communicating with the non-RT threads via a lock-free
ring-buffer.  The Hard-RT thread would be very well behaved and not do one iota
of memory allocation.  Conversely, the other threads, especially the GUI thread,
would.

The D spec says that all threads must pause when the GC runs, regardless of
which thread it runs in.  This seems to invalidate my design's RT-ness.  The
obvious solution of making the program multiprocess (barf) isn't too useful,
because that would make adjusting the RT thread's memory allocation a rather
convoluted process.  With the current design, I can adjust the RT thread's
memory structures in one of the non-RT threads, and protect the data with a
mutex that is try_locked().  A multi-process solution would be markedly uglier,
and I can't think of any method that would be 100% correct too boot; tickling
the GC seems inevitable.

Is there any way around this?  Is it even possible to write a GC that doesn't
stop up all threads?

I think i found in the manual that the GC cycle cannot start if no allocation is done anywhere in the code. So maybe you can allocate all memory you need before the RT condition starts. So with a careful design you can make sure the gc doesn't get called during RT critical time slices. if you need to allocate, you can disable gc before and enable after doing so. Or you can allocate with malloc, the gc will ignore that but then you have to clean up yourself. At a first glance it should not be very difficult to build hard RT conditions into your D-code, but how do you want to obtain hard RT conditions from your OS? Check this out: http://www.flounder.com/time.htm - if that is true you will have to boot your machine with your program, in order to obtain hard RT conditions.
Sep 09 2004
parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
WOB wrote:

 At a first glance it should not be very difficult to build hard RT
 conditions into your D-code, but how do you want to obtain hard RT
 conditions from your OS? Check this out:
 http://www.flounder.com/time.htm - if that is true you will have to
 boot your machine with your program, in order to obtain hard RT
 conditions.

Depends on whether "your OS" really refers to Windows as you silently assume. There is a number of operating systems out there that support hard RT. One example is RTLinux, a modification of the original Linux kernel. There even is a free version of that to check out: http://www.fsmlabs.com/products/openrtlinux/
Sep 09 2004
parent WOB <wb sapo.pt> writes:
On Fri, 10 Sep 2004 08:46:15 +0200, Norbert Nemec
<Norbert Nemec-online.de> wrote:

Depends on whether "your OS" really refers to Windows as you silently
assume. There is a number of operating systems out there that support hard
RT. 

Well I didn't silently assume Windows but yes I did silently assume any kind of general-purpose OS since there was also a reference to GUI threads in the original post, but no reference to RT constraints of the OS. But you are right, I should have written "... with your own program, or install a real time OS..."
Sep 10 2004
prev sibling next sibling parent Ben Hinkle <bhinkle4 juno.com> writes:
ninjadroid gazuga.net wrote:

 This is NOT flamebait!
 
 I like D.  I want the answer to the question posed in the subject of this
 post
 to be a hearty and resounding "yes!"  However, I have some concerns that
 such won't be the case.
 
 D is a GC'd language through and through, and I'm not interested in trying
 to
 extirpate that quality.  I imagine that if I wrote a program that didn't
 tickle the GC in any way, shape, or form (i.e. 100% static memory), then
 it would be
 Hard-RT compliant.  However, it probably wouldn't be terribly useful.
 
 The specific scenario I'm interested in is multi-threaded programming with
 one Hard-RT thread, communicating with the non-RT threads via a lock-free
 ring-buffer.  The Hard-RT thread would be very well behaved and not do one
 iota
 of memory allocation.  Conversely, the other threads, especially the GUI
 thread, would.
 
 The D spec says that all threads must pause when the GC runs, regardless
 of
 which thread it runs in.  This seems to invalidate my design's RT-ness. 
 The obvious solution of making the program multiprocess (barf) isn't too
 useful, because that would make adjusting the RT thread's memory
 allocation a rather
 convoluted process.  With the current design, I can adjust the RT thread's
 memory structures in one of the non-RT threads, and protect the data with
 a
 mutex that is try_locked().  A multi-process solution would be markedly
 uglier, and I can't think of any method that would be 100% correct too
 boot; tickling the GC seems inevitable.
 
 Is there any way around this?  Is it even possible to write a GC that
 doesn't stop up all threads?

sure - that wouldn't be hard at all. I would modify two files: std/thread.d and internal/gc/gcx.d To thread.d I'd add a flag indicating if the thread is not to be stopped during GC (and ignore such threads during pauseAll) and to gcx.d in the function fullCollect I'd weed out those threads that are flagged. It would be up to the programmer to make sure the non-GC thread doesn't contain any sole references to GC-managed data. It would probably be a while before anything like this made it into D proper - if at all because it is trusting the programmer quite a lot to mark a thread as invisible to the GC. Am I understanding the situation - or are there GC-managed objects that have the only live reference from the RT thread? -Ben
Sep 09 2004
prev sibling parent reply "Walter" <newshound digitalmars.com> writes:
<ninjadroid gazuga.net> wrote in message
news:chqq71$k0o$1 digitaldaemon.com...
 The D spec says that all threads must pause when the GC runs, regardless

 which thread it runs in.  This seems to invalidate my design's RT-ness.

 obvious solution of making the program multiprocess (barf) isn't too

 because that would make adjusting the RT thread's memory allocation a

 convoluted process.  With the current design, I can adjust the RT thread's
 memory structures in one of the non-RT threads, and protect the data with

 mutex that is try_locked().  A multi-process solution would be markedly

 and I can't think of any method that would be 100% correct too boot;

 the GC seems inevitable.

 Is there any way around this?  Is it even possible to write a GC that

 stop up all threads?

The gc only pauses all threads that it knows about. There's nothing stopping a D program from bypassing std.thread and creating a thread directly from the operating system APIs. That thread wouldn't be paused during a gc collection cycle, however, it would be incumbent upon the programmer to make *sure* that the unpaused thread does not hold any references to objects that *are not also held by any paused threads*. D, being a systems programming language, lets you cheat and break the rules when necessary.
Sep 09 2004
parent reply ninjadroid gazuga.net writes:
In article <chr729$uvr$1 digitaldaemon.com>, Walter says...
The gc only pauses all threads that it knows about. There's nothing stopping
a D program from bypassing std.thread and creating a thread directly from
the operating system APIs. That thread wouldn't be paused during a gc
collection cycle, however, it would be incumbent upon the programmer to make
*sure* that the unpaused thread does not hold any references to objects that
*are not also held by any paused threads*.

OK, so to rephrase for those of us (i.e. me) that don't follow this kind of stuff too well: If I create a thread via a direct call to the OS, the GC won't know about it. If the GC doesn't know about the thread, it won't stop it during a collection run. However, I need to then make sure that all the allocated memory the "renegade" thread refers to is also referenced by a thread that the GC *is* aware of; otherwise, the GC will think that memory is actually unused, and possibly make my life miserable as a result. Somewhat OT question: in a GC thread, will calls to malloc() register with the GC? That is, if you allocate memory with malloc(), you don't have to worry about the GC gobbling it up?
D, being a systems programming language, lets you cheat and break the rules
when necessary.

That is *music* to my ears!
Sep 10 2004
next sibling parent Sean Kelly <sean f4.ca> writes:
In article <chsal1$1guv$1 digitaldaemon.com>, ninjadroid gazuga.net says...
Somewhat OT question: in a GC thread, will calls to malloc() register with the
GC?  That is, if you allocate memory with malloc(), you don't have to worry
about the GC gobbling it up?

No. Though it's worth noting that most memory allocators have critical sections. It may be worth writing your own allocator or seeing if you can integrate an aftermarket version if you are concerned about this bottleneck.
That is *music* to my ears!

Where is BeOS when you need it? ;) Sean
Sep 10 2004
prev sibling parent Sha Chancellor <schancel pacific.net> writes:
In article <chsal1$1guv$1 digitaldaemon.com>, ninjadroid gazuga.net 
wrote:

 In article <chr729$uvr$1 digitaldaemon.com>, Walter says...
The gc only pauses all threads that it knows about. There's nothing stopping
a D program from bypassing std.thread and creating a thread directly from
the operating system APIs. That thread wouldn't be paused during a gc
collection cycle, however, it would be incumbent upon the programmer to make
*sure* that the unpaused thread does not hold any references to objects that
*are not also held by any paused threads*.

OK, so to rephrase for those of us (i.e. me) that don't follow this kind of stuff too well: If I create a thread via a direct call to the OS, the GC won't know about it. If the GC doesn't know about the thread, it won't stop it during a collection run. However, I need to then make sure that all the allocated memory the "renegade" thread refers to is also referenced by a thread that the GC *is* aware of; otherwise, the GC will think that memory is actually unused, and possibly make my life miserable as a result.

"it would be incumbent upon the programmer to make sure* that the unpaused thread does _NOT_ hold any references to objects that are not also held by any paused threads*."
 
 Somewhat OT question: in a GC thread, will calls to malloc() register with the
 GC?  That is, if you allocate memory with malloc(), you don't have to worry
 about the GC gobbling it up?
 
D, being a systems programming language, lets you cheat and break the rules
when necessary.

That is *music* to my ears!

Sep 12 2004