www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Standard Event Driven Object Library for D

reply Brian White <bcwhite pobox.com> writes:
For the last 15 years, I've been working with a custom C++ library for 
doing event-driven development.  Neither of D's "standard" libraries 
appear to support this kind of development and I was wondering if there 
is any interested in working on one.

That's not to say that you cannot do event-driven development given the 
current libraries, but it follows the form of:

- do a "select" (epoll, whatever)
- if socket "a" has a read event, do Xr
- if socket "a" has a write event, do Xw
- if socket "b" has a read event, do Yr
- etc.
- repeat

What I'm talking about is a library where all the objects handle events 
and in turn generate their own.  When you create a socket, it 
automatically registers itself with an event loop for that thread.  When 
data comes in, the socket object gets the event with no "user code" 
required.  The socket does what it needs for tracking the connection and 
then sends an event to (for example) the socket-stream.  It does what it 
needs and calls whatever object is controlling it.

In the end, it becomes easy to add self-contained services.  A simple 
HTTP server, for example, can be started with a set of files and 
absolutely no support from "main".  It doesn't have to be passed an open 
connection, a listening socket, or even a select-server.  In addition, 
the HTTP server itself can make choices about how it receives events, 
turning them on or off as necessary, or perhaps going so far as spawing 
new threads.  When a new connection comes in, the accepting of a new 
socket bound to a connection-handler object means that the connection 
handler is going to get the events associated with that socket.

Now it doesn't seem at first glance that this gains you much.  If the 
HTTP server is getting notices directly from a select-server, then it 
can do a read (or write, as appropriate) to the correct socket and 
process the results.  That's self-contained, too.  But when the events 
always flow from the bottom-up, here are some other things you could get:

- Buffering:  The socket stream (indeed, the base stream itself) can do 
read/write bufferring with no support from the top.  In the case of a 
write buffer, in can enable/disable write events when needed with no 
invervention from the controlling object.  And, you only need to write 
this code once to have it work with HTTP and all other socket services.

- Specialization:  A select server gives you "read" and "write" events. 
  A socket can generate "read", "write", "connected", "closed", and 
"broken" events.  Why have each network service have to differentiate 
these independently?

- Extensibility:  A new feature of the HTTP service is now to accept 
SIGHUP to reload a set of files.  Without touching one line of "main", 
you simply have the HTTPserver object make a request for events from the 
global signal handler.  Want to time-out connections?  Have the socket 
request events from a timer module and notify the server if a time-out 
occurs.

- Parallelism:  It becomes easier (not that multi-threading is ever 
easy) to run different objects on different execute threads.  Since 
there is no longer a global event loop created/instantiated by "main", 
the library can make its own decisions (with restrictions, of course).

In my C++ code, I did all this with callbacks instead of posted events 
because, well, it's more efficient and makes better sense than a single 
event handling function per object with a big switch statement.  D's 
delegates and interfaces should make this even easier.


Now it's not to say that this method of programming is without it's 
difficulties.  You can get loops.  For example: a socket gets a read 
event, passes it all the way up to the HTTPserver object which, for 
whatever reason, decides to close the connection.  The "close" call goes 
down the stack, affects the underlying path, deregisters it from the 
select-server, and returns... all the way up to the HTTPserver and then 
back down the event callbacks to the select-server where it generated 
the read event.  The socket was open and valid when it started the 
callback but now it's closed.  You have to plan for this.

The other big problem is that you can't do this with an existing 
library.  It has be built-in from the ground up.  Doing this in the D 
library would require a new socket class, a new stream class, etc., etc. 
  It doesn't have to replace the existing one but it would be parallel.


What do others think of this?

-- Brian
Apr 01 2008
next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Brian White wrote:
 What do others think of this?
 
 -- Brian

% toc: % events % events in stdlib \section{events} I'm also interested in the subject, though I haven't had a chance to work on it. (I'm trying to get a full agile setup in D; I've done mock objects, an xUnit, and dependency injection system so far. Events are on my list, but not yet accomplished.) What sort of API would you want? Jeremy Miller's event system (in C#) looks like: interface IListener<T> { void Handle(T subject); } class EventPublisher { public void AddListener(object o){} public void Raise<T>(T subject){} } That does a simple O(n) loop on each event to see if any of the registered objects can be converted to an IListener<T>. This is convenient because any class can listen to an arbitrary number of events. It is inconvenient that publishing an event is linear in the number of total listeners rather than the number of listeners for that event. You probably could do some reflection to find which events an object listens to, though. (I wouldn't particularly want to do that in D1, though.) Also, you can't use structs to receive events, if you translate it to D. The other issue is that you need to create a class (or probably struct) for every event you use. CAB and Ayende Rahien's event broker both use strings to describe which event you care about. This means you have fewer UDTs, but then you're using strings, which have zero validation. It also makes it a bit more ugly to provide arguments. Jeremy Miller's system has the best validation -- it's your code that won't compile if you don't listen to the event properly. The worst you could do is send the wrong event and have the wrong stuff happen rather than send an event and have nothing happen -- the former is probably a bit easier to debug. Using strings as event URIs has a benefit that you can use hashing. However, it requires more configuration. Events are easier to work with if you use hooks into a dependency injection tool to wire them up. There is, at last count, exactly one dependency injection library for D, and that's mine. Dconstructor doesn't currently support interceptors that run after the construction of an object. But for event listeners, you could still put a line in the constructor: // if you're using dependency injection this (EventRegistrar events) { events ~= this; } // if you're using the singleton pattern this () { EventRegistrar.instance() ~= this; } \section{events in stdlib} I'm not sure that sockets with events are appropriate for the standard library. You'd need, at a minimum, one extra thread that the sockets operate on, unless you still had a socket set that you polled. This means that any application that used a socket would need to concern itself with thread safety. This would be okay if you had two classes for sockets in the stdlib, say PollingSocket and EventSocket. But still, I would want to control my threads. You could instead have a static method on EventSocket that polls each EventSocket (or uses a SocketSet to do so), and if any socket that has something to report, it will publish the appropriate event. Still, I'm not sure something like this belongs in the standard library quite yet.
Apr 01 2008
parent reply Brian White <bcwhite pobox.com> writes:
 I'm also interested in the subject, though I haven't had a chance to 
 work on it. (I'm trying to get a full agile setup in D; I've done mock 
 objects, an xUnit, and dependency injection system so far. Events are on 
 my list, but not yet accomplished.) What sort of API would you want?

The event interface you describe is similar (the same?) as the signals module included, but that's not what I'm thinking about. Instead, I'm leaning towards something like: interface PathHandlerInterface { bool PathReadHandler(int path); bool PathWriteHandler(int path); } ...and... interface SocketHandlerInterface { bool SocketSendHandler(Socket s); bool SocketRecvHandler(Socket s); bool SocketConnectedHandler(Socket s); bool SocketBrokenHandler(Socket s): bool SocketClosedHandler(Socket s); } ...and... interface StreamHandlerInterface { bool StreamReadHandler(stream s); bool StreamWriteHandler(stream s); bool StreamOpenedHandler(stream s); bool StreamClosedHandler(stream s); } interface SocketStreamHandlerInterface : StreamHandlerInterface { bool StreamBrokenHandler(stream s); } ...and... class HttpServer : SocketStreamHandlerInterface { ... } The idea, then, is that (for example) an HTTPserver object would derive from SocketStreamHandlerInterface for handing events. The SocketStream class would have a pointer to an attached class and, if non-null, would call those methods off of it whenever it detected the stated condition. That detection would generally be done in Socket*Handler implementations which were called by the attached Socket class. It, in turn, probably called that function from within a Stream*Handler method called by the base Thread object managing the events. As an example... - An EvThread instance does a select and eventually gets a read event on path #N. It does some work and then calls handler.PathReadHandler(pathno). - The EvSocket object that was called as a result does some processing and eventually calls handler.SocketReadHandler(this). - The EvSocketStream object that was called as a result does some processing and eventually calls handler.SocketStreamReadHandler(this); - The EvHttpServer object that was called as a result reads the pending data, does its magic, writes a result, and returns "true" to say it handled the event and wants more of them. - The EvSocketStream object thinks all is good in the world and also returns "true". - The EvSocket object thinks all is good in the world and also returns "true". - The EvThread object things all is good in the world and starts another select loop. Had the event not been handled (returned "false"), it would have disabled read events on that path and continued on. -- Brian
Apr 01 2008
parent reply Christopher Wright <dhasenan gmail.com> writes:
Brian White wrote:
 As an example...
 
 - An EvThread instance does a select and eventually gets a read event on 
 path #N.  It does some work and then calls handler.PathReadHandler(pathno).
 
 - The EvSocket object that was called as a result does some processing 
 and eventually calls handler.SocketReadHandler(this).
 
 - The EvSocketStream object that was called as a result does some 
 processing and eventually calls handler.SocketStreamReadHandler(this);
 
 - The EvHttpServer object that was called as a result reads the pending 
 data, does its magic, writes a result, and returns "true" to say it 
 handled the event and wants more of them.
 
 - The EvSocketStream object thinks all is good in the world and also 
 returns "true".
 
 - The EvSocket object thinks all is good in the world and also returns 
 "true".
 
 - The EvThread object things all is good in the world and starts another 
 select loop.
 
 Had the event not been handled (returned "false"), it would have 
 disabled read events on that path and continued on.
 
 -- Brian

I'm seeing two types of events: notifications and responsibilities. A notification just says "This happened, and I don't care if anyone reacts to it." std.signals does okay with that, except for a few bits of ugliness and the fact that it places undesirable dependencies between objects. A responsibility says "This happened, and someone (but only one) should take care of it." It might be an error if nobody handles it. Something can take a look at a responsibility before determining that it's somebody else's problem. However, I don't see the utility in automatically removing a listener when it claims that it cannot handle a particular responsibility. Not handling a particular message and not handling any further messages are different use cases.
Apr 01 2008
parent reply Brian White <bcwhite pobox.com> writes:
 I'm seeing two types of events: notifications and responsibilities.
 
 A notification just says "This happened, and I don't care if anyone 
 reacts to it." std.signals does okay with that, except for a few bits of 
 ugliness and the fact that it places undesirable dependencies between 
 objects.
 
 A responsibility says "This happened, and someone (but only one) should 
 take care of it." It might be an error if nobody handles it. Something 
 can take a look at a responsibility before determining that it's 
 somebody else's problem.

That's a good distinction. The word "event" is too loose.
 However, I don't see the utility in automatically removing a listener 
 when it claims that it cannot handle a particular responsibility. Not 
 handling a particular message and not handling any further messages are 
 different use cases.

You could be right. That's why I want to get other people involved; I can't see everything. Here was my reasoning... When doing embedded controller work, I used a very similar design for passing data up a TCP/IP stack to the app. During profiling, I found that a significant amount of the CPU time was being spent in the dispatching of i/o events that just got ignored. By changing the event handlers to say "thanks, but don't send me any more", I was able to save something like 10-15% CPU time. (Profiling never shows you what you expect. :-) In the case of a C++ program in a proper operating system, think of the "write" event from a select call. If you leave it active but write nothing, you'll just get the event again and again and again. You'll spin CPU instead of letting the OS notify you. There are other solutions, of course. In my C++ library, you had to make a call to clear the "write callback" when you didn't want such events. That would work it's way down the stack of objects until it cleared the respective bit from the "write" fd_set. This ended up being somewhat clumsy to implement though because objects are responsible for themselves, it was never anything the library user had to worry about. Instead, I thought I'd take a simpler approach of just requesting "all or nothing" and letting the system take care of turning off those that aren't wanted when they occurred. I figured it would also be beneficial to minimize the number of potential loops in the object stack. If calling "DisableEvent" from within an event handler, you run the risk of that method generating some different event and calling back up the stack yet again. By returning the status, then any other events generated aren't a loop. In the presence of a proper event queue, loops never happens because the next item on the queue doesn't get processed until the previous one completes. Using callbacks, however, this is not the case as every "event" is always processed directly and immediately. -- Brian
Apr 01 2008
parent reply Christopher Wright <dhasenan gmail.com> writes:
Brian White wrote:
 I'm seeing two types of events: notifications and responsibilities.

 A notification just says "This happened, and I don't care if anyone 
 reacts to it." std.signals does okay with that, except for a few bits 
 of ugliness and the fact that it places undesirable dependencies 
 between objects.

 A responsibility says "This happened, and someone (but only one) 
 should take care of it." It might be an error if nobody handles it. 
 Something can take a look at a responsibility before determining that 
 it's somebody else's problem.

That's a good distinction. The word "event" is too loose.
 However, I don't see the utility in automatically removing a listener 
 when it claims that it cannot handle a particular responsibility. Not 
 handling a particular message and not handling any further messages 
 are different use cases.

You could be right. That's why I want to get other people involved; I can't see everything. Here was my reasoning... When doing embedded controller work, I used a very similar design for passing data up a TCP/IP stack to the app. During profiling, I found that a significant amount of the CPU time was being spent in the dispatching of i/o events that just got ignored. By changing the event handlers to say "thanks, but don't send me any more", I was able to save something like 10-15% CPU time. (Profiling never shows you what you expect. :-)

Interesting. I suppose in GUI design, one widget will always or never want to handle a responsibility. You could return an enum value PassOn, Stop, or SendNoMore. And then the library would want to determine which to return, and just accept a void delegate for the event handler.
 In the case of a C++ program in a proper operating system, think of the 
 "write" event from a select call.  If you leave it active but write 
 nothing, you'll just get the event again and again and again.  You'll 
 spin CPU instead of letting the OS notify you.
 
 There are other solutions, of course.  In my C++ library, you had to 
 make a call to clear the "write callback" when you didn't want such 
 events.  That would work it's way down the stack of objects until it 
 cleared the respective bit from the "write" fd_set.  This ended up being 
 somewhat clumsy to implement though because objects are responsible for 
 themselves, it was never anything the library user had to worry about.

A bit easier to manage if you're using singletons, though that doesn't work in many cases. Or you could pass in whatever created the event along with the event data.
 Instead, I thought I'd take a simpler approach of just requesting "all 
 or nothing" and letting the system take care of turning off those that 
 aren't wanted when they occurred.
 
 I figured it would also be beneficial to minimize the number of 
 potential loops in the object stack.  If calling "DisableEvent" from 
 within an event handler, you run the risk of that method generating some 
 different event and calling back up the stack yet again.  By returning 
 the status, then any other events generated aren't a loop.

I've had some issues with that in the past. I don't think there is a simple solution. Though for GUI stuff, you could often use data binding for simple cases.
 In the presence of a proper event queue, loops never happens because the 
 next item on the queue doesn't get processed until the previous one 
 completes.  Using callbacks, however, this is not the case as every 
 "event" is always processed directly and immediately.

What prevents you from adding to the queue an event that will cause this event to be added to the queue again?
 -- Brian

Apr 02 2008
parent Brian White <bcwhite pobox.com> writes:
 When doing embedded controller work, I used a very similar design for 
 passing data up a TCP/IP stack to the app.  During profiling, I found 
 that a significant amount of the CPU time was being spent in the 
 dispatching of i/o events that just got ignored.  By changing the 
 event handlers to say "thanks, but don't send me any more", I was able 
 to save something like 10-15% CPU time.  (Profiling never shows you 
 what you expect.  :-)

Interesting. I suppose in GUI design, one widget will always or never want to handle a responsibility. You could return an enum value PassOn, Stop, or SendNoMore. And then the library would want to determine which to return, and just accept a void delegate for the event handler.

I tried using it under Windows once, setting callback functions for various events. It worked okay, but didn't allow for multiple watchers. I think for the GUI it's better to use a signals & slots model. It provides more flexibility. I/O, on the other hand, usually has just one chain of objects.
 In the case of a C++ program in a proper operating system, think of 
 the "write" event from a select call.  If you leave it active but 
 write nothing, you'll just get the event again and again and again.  
 You'll spin CPU instead of letting the OS notify you.

 There are other solutions, of course.  In my C++ library, you had to 
 make a call to clear the "write callback" when you didn't want such 
 events.  That would work it's way down the stack of objects until it 
 cleared the respective bit from the "write" fd_set.  This ended up 
 being somewhat clumsy to implement though because objects are 
 responsible for themselves, it was never anything the library user had 
 to worry about.

A bit easier to manage if you're using singletons, though that doesn't work in many cases. Or you could pass in whatever created the event along with the event data.

I always include the generator object in the call. That allows a single object to accept from multiple sources, like an HTTPserver with multiple open connections.
 Instead, I thought I'd take a simpler approach of just requesting "all 
 or nothing" and letting the system take care of turning off those that 
 aren't wanted when they occurred.

 I figured it would also be beneficial to minimize the number of 
 potential loops in the object stack.  If calling "DisableEvent" from 
 within an event handler, you run the risk of that method generating 
 some different event and calling back up the stack yet again.  By 
 returning the status, then any other events generated aren't a loop.

I've had some issues with that in the past. I don't think there is a simple solution. Though for GUI stuff, you could often use data binding for simple cases.
 In the presence of a proper event queue, loops never happens because 
 the next item on the queue doesn't get processed until the previous 
 one completes.  Using callbacks, however, this is not the case as 
 every "event" is always processed directly and immediately.

What prevents you from adding to the queue an event that will cause this event to be added to the queue again?

Nothing, but there are differences. Any given event gets fully processed before the next one. With callbacks, you can get one within another. In the case of an event adding itself, you effectively have an infinite loop. In the case of a callback calling itself, you have a stack overflow. -- Brian
Apr 02 2008
prev sibling next sibling parent reply Leandro Lucarella <llucax gmail.com> writes:
Brian White, el  1 de abril a las 13:12 me escribiste:
 For the last 15 years, I've been working with a custom C++ library for doing
event-driven 
 development.  Neither of D's "standard" libraries appear to support this kind
of 
 development and I was wondering if there is any interested in working on one.

I guess you're looking for something D-native and more high-level, but I guess my D "bindings" for libev[1] could be of your interest. For know, I have just a git repository[2], but there are some examples and tests in the repository you can find useful as "documentation". [1] http://software.schmorp.de/pkg/libev.html [2] http://git.llucax.com.ar/?p=software/ev.d.git;a=summary -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- no longer afraid of the dark or midday shadows nothing so ridiculously teenage and desperate, nothing so childish - at a better pace, slower and more calculated, no chance of escape,
Apr 01 2008
parent Brian White <bcwhite pobox.com> writes:
 I guess you're looking for something D-native and more high-level, but
 I guess my D "bindings" for libev[1] could be of your interest. For know,
 I have just a git repository[2], but there are some examples and tests in
 the repository you can find useful as "documentation".

I'm looking D-native, yes, but really pretty low-level. The generic event throw/catch model (aka "signals & slots") works great for generic concepts like "the user just selected a new color" because many independent objects may be interested in it, or none may be interested. It's very arbitrary. When it comes to I/O, however, things behave somewhat differently. There is typically a chain of objects that are interested and the ability to pass an event up that chain, modifying it or altering it as appropriate, works very well. This has been my experience with my C++ library, anyway. -- Brian
Apr 01 2008
prev sibling parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Brian White wrote:
 For the last 15 years, I've been working with a custom C++ library for 
 doing event-driven development.  Neither of D's "standard" libraries 
 appear to support this kind of development and I was wondering if there 
 is any interested in working on one.
 
 That's not to say that you cannot do event-driven development given the 
 current libraries, but it follows the form of:
 
 - do a "select" (epoll, whatever)
 - if socket "a" has a read event, do Xr
 - if socket "a" has a write event, do Xw
 - if socket "b" has a read event, do Yr
 - etc.
 - repeat
 
 What I'm talking about is a library where all the objects handle events 
 and in turn generate their own.  When you create a socket, it 
 automatically registers itself with an event loop for that thread.  When 
 data comes in, the socket object gets the event with no "user code" 
 required.  The socket does what it needs for tracking the connection and 
 then sends an event to (for example) the socket-stream.  It does what it 
 needs and calls whatever object is controlling it.
 
 In the end, it becomes easy to add self-contained services.  A simple 
 HTTP server, for example, can be started with a set of files and 
 absolutely no support from "main".  It doesn't have to be passed an open 
 connection, a listening socket, or even a select-server.  In addition, 
 the HTTP server itself can make choices about how it receives events, 
 turning them on or off as necessary, or perhaps going so far as spawing 
 new threads.  When a new connection comes in, the accepting of a new 
 socket bound to a connection-handler object means that the connection 
 handler is going to get the events associated with that socket.
 
 Now it doesn't seem at first glance that this gains you much.  If the 
 HTTP server is getting notices directly from a select-server, then it 
 can do a read (or write, as appropriate) to the correct socket and 
 process the results.  That's self-contained, too.  But when the events 
 always flow from the bottom-up, here are some other things you could get:
 
 - Buffering:  The socket stream (indeed, the base stream itself) can do 
 read/write bufferring with no support from the top.  In the case of a 
 write buffer, in can enable/disable write events when needed with no 
 invervention from the controlling object.  And, you only need to write 
 this code once to have it work with HTTP and all other socket services.
 
 - Specialization:  A select server gives you "read" and "write" events. 
  A socket can generate "read", "write", "connected", "closed", and 
 "broken" events.  Why have each network service have to differentiate 
 these independently?
 
 - Extensibility:  A new feature of the HTTP service is now to accept 
 SIGHUP to reload a set of files.  Without touching one line of "main", 
 you simply have the HTTPserver object make a request for events from the 
 global signal handler.  Want to time-out connections?  Have the socket 
 request events from a timer module and notify the server if a time-out 
 occurs.
 
 - Parallelism:  It becomes easier (not that multi-threading is ever 
 easy) to run different objects on different execute threads.  Since 
 there is no longer a global event loop created/instantiated by "main", 
 the library can make its own decisions (with restrictions, of course).
 
 In my C++ code, I did all this with callbacks instead of posted events 
 because, well, it's more efficient and makes better sense than a single 
 event handling function per object with a big switch statement.  D's 
 delegates and interfaces should make this even easier.
 
 
 Now it's not to say that this method of programming is without it's 
 difficulties.  You can get loops.  For example: a socket gets a read 
 event, passes it all the way up to the HTTPserver object which, for 
 whatever reason, decides to close the connection.  The "close" call goes 
 down the stack, affects the underlying path, deregisters it from the 
 select-server, and returns... all the way up to the HTTPserver and then 
 back down the event callbacks to the select-server where it generated 
 the read event.  The socket was open and valid when it started the 
 callback but now it's closed.  You have to plan for this.
 
 The other big problem is that you can't do this with an existing 
 library.  It has be built-in from the ground up.  Doing this in the D 
 library would require a new socket class, a new stream class, etc., etc. 
  It doesn't have to replace the existing one but it would be parallel.
 
 
 What do others think of this?
 
 -- Brian

I was thinking about porting SEDA, but Descent is taking all my programming time up right now.
Apr 01 2008
parent Robert Fraser <fraserofthenight gmail.com> writes:
Robert Fraser wrote:
 I was thinking about porting SEDA, but Descent is taking all my 
 programming time up right now.

Should have provided the link: SEDA website: http://www.eecs.harvard.edu/~mdw/proj/seda/
Apr 01 2008