www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Is there interest in a std.http?

reply "Tyler Jameson Little" <beatgammit gmail.com> writes:
I'd like to see an HTTP module in Phobos, but I wanted to gauge 
interest first and see if this has been discussed before.

An example of how I would like to interface with it (for creating 
a server):

     interface HTTPHandler {
         void serveHTTP(Request, Response);
     }

     class CustomHandler : HTTPHandler {
         ....
         void serveHTTP(Request req, Response res) {
         }
     }

     auto handler = new CustomHandler(); // implements http.Handler
     auto server = new HttpServer(handler);
     server.listen("0.0.0.0", 80);

As long as serveHttp() is thread safe, the http server could be 
concurrent or evented (with libev or similar) and the custom 
handler code wouldn't have to change.

I'm willing to put in the lion's share of the work (I've already 
written a bunch of it), but I'd naturally like to get some 
community input so I don't go in a completely wrong direction. 
I'm thinking of emulating Go's http library 
(http://golang.org/pkg/net/http), but of course D style.

I think the following are necessary:

* Access to underlying TcpSocket (for protocol upgrades, like 
WebSocket)
* HTTP body is a stream
* Simple HTTP requests
* Errors can be recovered from (allow user-defined error 
handlers):
* User settable size limits
   * Size of request line (1024 bytes by default)
   * Size of each header (1024 bytes by default)
   * Total size of header block (4096 bytes by default)

So, basically I'm looking for two things:

1. Interest level: willing to writ code, willing to test, want to 
use
2. Suggestions for the API/behavior
Nov 19 2012
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little 
wrote:
 I'd like to see an HTTP module in Phobos, but I wanted to gauge 
 interest first and see if this has been discussed before.

I've been asked to put my cgi.d in phobos before (which includes a http server as well as a cgi, fastcgi, and scgi implementation of its common interface), so there's probably some interest. I haven't put mine in just because I'm not particularly motivated to go through the red tape. The file is cgi.d in here: https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff Feel free to take whatever you want from it. There's also an extremely minimal http.d file in there, which is a very simple http client. (I didn't go far with this though because I tend to use curl for most my work anyway.) The way I did it is with a function: void handleRequest(Cgi cgi) { } And then there's a mixin called GenericMain that calls your function. mixin GenericMain!handleRequest; Depending on how you compile it, the main function constructs the cgi object differently. -version=fastcgi means it is plugged into a FastCGI loop. version=scgi ditto. -version=embedded_httpd includes a simple little threaded http server. No version builds it as a standard CGI binary, one process per request. However it is compiled though, the Cgi methods all remain the same, so your function shouldn't have to change.
 * Access to underlying TcpSocket (for protocol upgrades, like 
 WebSocket)

My cgi.d I don't think can do this yet. I've been thinking about adding it though, but haven't.
 * HTTP body is a stream

class Cgi has a method onRequestBodyDataReceived and one handleIncomingDataChunk which can be overridden by subclasses to do this kind of thing. It's a bit of a pain though, I made it thinking you'd want to handle it after it is all loaded.
 * Simple HTTP requests

This is where I think my cgi.d is a winner because most the data is available so easily cgi.get["foo"] returns a string cgi.getArray["foo"] returns an array of strings ditto for post cgi.request("foo", 0) returns with a default value, converting types as requested There's also members like cookie, queryString, host, etc., that gives nice access to other headers.
 * Errors can be recovered from (allow user-defined error 
 handlers):
 * User settable size limits

eh, I'm partially there on these. There's some constructor args you can play with and some degree of exception catching but probably not fully what you have in mind.
Nov 19 2012
prev sibling next sibling parent "Tyler Jameson Little" <beatgammit gmail.com> writes:
 I've been asked to put my cgi.d in phobos before (which 
 includes a http server as well as a cgi, fastcgi, and scgi 
 implementation of its common interface), so there's probably 
 some interest.

 I haven't put mine in just because I'm not particularly 
 motivated to go through the red tape.

 The file is cgi.d in here:
 https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff

Awesome. I assume this hasn't gone through rigorous testing, but has it been used in production?
 * Errors can be recovered from (allow user-defined error 
 handlers):
 * User settable size limits

eh, I'm partially there on these. There's some constructor args you can play with and some degree of exception catching but probably not fully what you have in mind.

I'm thinking of something a little more sophisticated than exceptions. Exceptions unwind the stack, so you can't just continue where you left off. How do you feel about function pointer callbacks? The http parser in node.js (https://github.com/joyent/http-parser) takes a struct of function pointers to handle errors/events. When the parser encounters a recoverable error (max header length reached), the programmer could opt to ignore it and continue the parse. Unrecoverable errors could throw exceptions (like trying to parse garbage data). If an error is recovered, it would just continue as if nothing happened. Of course, implementing this would increase complexity in the parser, but I think it's justified. Thoughts?
Nov 19 2012
prev sibling next sibling parent "Pragma Tix" <bizprac orange.fr> writes:
Before you spend too much time :
https://github.com/rejectedsoftware/vibe.d/tree/master/source/vibe/http



On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little 
wrote:
 I'd like to see an HTTP module in Phobos, but I wanted to gauge 
 interest first and see if this has been discussed before.

 An example of how I would like to interface with it (for 
 creating a server):

     interface HTTPHandler {
         void serveHTTP(Request, Response);
     }

     class CustomHandler : HTTPHandler {
         ....
         void serveHTTP(Request req, Response res) {
         }
     }

     auto handler = new CustomHandler(); // implements 
 http.Handler
     auto server = new HttpServer(handler);
     server.listen("0.0.0.0", 80);

 As long as serveHttp() is thread safe, the http server could be 
 concurrent or evented (with libev or similar) and the custom 
 handler code wouldn't have to change.

 I'm willing to put in the lion's share of the work (I've 
 already written a bunch of it), but I'd naturally like to get 
 some community input so I don't go in a completely wrong 
 direction. I'm thinking of emulating Go's http library 
 (http://golang.org/pkg/net/http), but of course D style.

 I think the following are necessary:

 * Access to underlying TcpSocket (for protocol upgrades, like 
 WebSocket)
 * HTTP body is a stream
 * Simple HTTP requests
 * Errors can be recovered from (allow user-defined error 
 handlers):
 * User settable size limits
   * Size of request line (1024 bytes by default)
   * Size of each header (1024 bytes by default)
   * Total size of header block (4096 bytes by default)

 So, basically I'm looking for two things:

 1. Interest level: willing to writ code, willing to test, want 
 to use
 2. Suggestions for the API/behavior

Nov 19 2012
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little 
wrote:
 * HTTP body is a stream

No Content-Size, no multiple requests per connection (unless you use chunked encoding?).
 * User settable size limits
   * Size of request line (1024 bytes by default)
   * Size of each header (1024 bytes by default)
   * Total size of header block (4096 bytes by default)

I think this is pointless. Use an appender and the RFC limits. Unless you are serving VERY simple pages, the cost of a few allocations for handling the HTTP protocol's overhead will not be noticeable compared to the application's. (Slowloris is something to keep in mind, though.) My HTTP library, which is used for this forum and several other "production" projects, is here: https://github.com/CyberShadow/ae The main problem with getting it into Phobos is that it's tied to a lot of other related code, such as the asynchronous sockets module, the unmanaged memory wrapper type, etc. Also, what about vibe.d?
Nov 19 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 19 November 2012 at 23:57:35 UTC, Tyler Jameson Little 
wrote:
 Awesome. I assume this hasn't gone through rigorous testing, 
 but has it been used in production?

I've been using it for almost all my work apps for the last... I think three years now. None of them so far have huge userbases, but it has been in steady use for a while. The others in this thread have pointed out the other options that have some users: Vladimir's library that runs the forum, and vibe.d that seems to be the most popular. I haven't used either of them though.
 How do you feel about function pointer callbacks?

We could also do them as virtual functions... that's a little easier in the setup I have because the constructor arguments are messy, whereas subclassing is a lot simpler. But yeah, it sounds like a decent plan.
Nov 19 2012
prev sibling next sibling parent "Tyler Jameson Little" <beatgammit gmail.com> writes:
 * HTTP body is a stream

No Content-Size, no multiple requests per connection (unless you use chunked encoding?).

Not sure what you mean. I meant incoming stream. There would be a request object with access to all other headers, it just wouldn't be read in until the user actually wanted it.
 * User settable size limits
  * Size of request line (1024 bytes by default)
  * Size of each header (1024 bytes by default)
  * Total size of header block (4096 bytes by default)

I think this is pointless. Use an appender and the RFC limits. Unless you are serving VERY simple pages, the cost of a few allocations for handling the HTTP protocol's overhead will not be noticeable compared to the application's. (Slowloris is something to keep in mind, though.)

I just made up those limits, but the point is I'd like the user to be able to tweak those. The default will work for most people though.
 My HTTP library, which is used for this forum and several other 
 "production" projects, is here: 
 https://github.com/CyberShadow/ae
 The main problem with getting it into Phobos is that it's tied 
 to a lot of other related code, such as the asynchronous 
 sockets module, the unmanaged memory wrapper type, etc.

Cool, I didn't know that this was served by D code. I'll take a look at it.
 Also, what about vibe.d?

vibe.d does a lot of things, and it probably does those things very well. It seems well maintained, and generally a good project. I think that a lot of the things it does well could be brought into the standard library. As this thread pointed out, there are several HTTP parsers floating around out there. For some, vibe.d might not be a perfect fit for whatever reason, but everyone can benefit from a simple HTTP library. Maybe it shouldn't be as high-level as Go's http library, but it should at least make writing a simple HTTP server trivial. Would a minor refactor of vibe.d be acceptable? This is pretty much what I'm looking for: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d Except that I'd remove some parts of it, like JSON parsing. I think that vibe.d could benefit from moving some of the code there into Phobos. I guess it comes down to whether it makes sense to make a standard HTTP library.
Nov 19 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/19/12 3:38 PM, Tyler Jameson Little wrote:
 I'd like to see an HTTP module in Phobos, but I wanted to gauge interest
 first and see if this has been discussed before.

I can say the following. We sorely need a server expert on board with the time and inclination to write a good server-side networking framework. We have a couple, but for various reasons I have been unable to convince them (through newsgroup and private exchanges) to invest time in such an endeavor. That's mostly because lack of time - speaking for myself, I wish I found the time to learn about all that, which as far as I understand gravitates around notions such as asynchronous I/O, libevent/libevt, select(), and the such. I've never used them so I'd need to start from scratch. We really need an expert. Look at the Go programming language - it's not remarkable, but it benefits of full-time dedication of experts in server-side programming, so it has acquired excellent library support for networking servers. And they milk that for all it's worth: any discussion, article, or blog post about Go gravitates toward the five-lines HTTP server with the same implacable reach as conversations with ideologists, which inevitably converge towards their ideological stronghold. Andrei
Nov 19 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/19/12 11:26 PM, Andrei Alexandrescu wrote:
 On 11/19/12 3:38 PM, Tyler Jameson Little wrote:
 I'd like to see an HTTP module in Phobos, but I wanted to gauge interest
 first and see if this has been discussed before.

I can say the following. We sorely need a server expert on board with the time and inclination to write a good server-side networking framework.

To clarify (lest this would be interpreted as excluding vibe.d from the category of good server-side networking frameworks): ... for inclusion in the standard library. Andrei
Nov 19 2012
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/20/2012 8:26 AM, Andrei Alexandrescu пишет:
 On 11/19/12 3:38 PM, Tyler Jameson Little wrote:
 I'd like to see an HTTP module in Phobos, but I wanted to gauge interest
 first and see if this has been discussed before.

I can say the following. We sorely need a server expert on board with the time and inclination to write a good server-side networking framework. We have a couple, but for various reasons I have been unable to convince them (through newsgroup and private exchanges) to invest time in such an endeavor. That's mostly because lack of time - speaking for myself, I wish I found the time to learn about all that, which as far as I understand gravitates around notions such as asynchronous I/O, libevent/libevt, select(), and the such. I've never used them so I'd need to start from scratch.

Having done quite some work on this before - both with frameworks and barehanded. The recipe for scalable and fast server itself is not that hard: asynchronous I/O + event-based notification + separate thread pool for user-defined handlers Asynchronous I/O is well understood to have far less overhead then thread per client/descriptor model. Both in terms of memory usage and time spent on the context switching. Event based notification can be traded for select() polling but then you'd waste a lot of cycles (in kernel mode) walking through huge descriptor sets when number of simultaneous clients is high enough. A separate "worker" thread pool insures your event-loop is not ever blocked thus insuring responsiveness w.r.t dispatching I/O requests. Arguably this could be an opt-in. If this combo is implemented even half-decently it should give us great speed. The fine-tuning and topping the performance however is the other 70% of work and that makes the real difference. The devil is in the details. Not to mention fiddling with socket options and imposing limits on transfer rate/timeouts etc. I currently had to work with Java (a painful journey after D) to create a couple of server components and surprisingly found the Netty project to be quite nice. Though being overly Java-escue it has some great ideas (aside from the above basic concepts): - The composable pipeline abstraction that allows passing user data through a series of transformations seamlessly. Think the stack of TCP|HTTP|encryption/compression or TCP|framing|ProtocolBuffers etc. - Flexible and zero-copy (quite hard for Java I guess) buffer abstraction. It reminds the power of ranges but in a heck of a more limited scale - it's more about chaining and slicing. These limitations are mostly because of the way the OS works.
 We really need an expert. Look at the Go programming language - it's not
 remarkable, but it benefits of full-time dedication of experts in
 server-side programming, so it has acquired excellent library support
 for networking servers.

Can help with a proper asynchronous (network) I/O support on Windows. I doubt I'll find time to flesh out the whole design though. What it irks me that I haven't seen a good enough backend for Windows in the open source (of those I've peeked at the source of). A lot of event-based backends seem to just ignore this OS or fallback to select-loop. Meanwhile M$ introduced the new RIO socket API targeted at high-load servers. Was planing to try it out since the day it was introduced.
 And they milk that for all it's worth: any
 discussion, article, or blog post about Go gravitates toward the
 five-lines HTTP server with the same implacable reach as conversations
 with ideologists, which inevitably converge towards their ideological
 stronghold.

I see no reason we can't beat them in their favorite game. -- Dmitry Olshansky
Nov 20 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/20/2012 11:45 PM, Dmitry Olshansky пишет:
[snip]


 I currently had to work with Java (a painful journey after D) to create
 a couple of server components and surprisingly found the Netty project
 to be quite nice.

The obligatory link: https://netty.io/ -- Dmitry Olshansky
Nov 20 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-11-19 21:38, Tyler Jameson Little wrote:

 I think the following are necessary:

 * Access to underlying TcpSocket (for protocol upgrades, like WebSocket)
 * HTTP body is a stream
 * Simple HTTP requests
 * Errors can be recovered from (allow user-defined error handlers):
 * User settable size limits
    * Size of request line (1024 bytes by default)
    * Size of each header (1024 bytes by default)
    * Total size of header block (4096 bytes by default)

* HTTPS -- /Jacob Carlborg
Nov 19 2012
prev sibling next sibling parent "Nicolas Sicard" <dransic gmail.com> writes:
On Tuesday, 20 November 2012 at 03:49:57 UTC, Tyler Jameson 
Little wrote:
 Would a minor refactor of vibe.d be acceptable? This is pretty 
 much what I'm looking for:

 https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d
 https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d

 Except that I'd remove some parts of it, like JSON parsing.

 I think that vibe.d could benefit from moving some of the code 
 there into Phobos. I guess it comes down to whether it makes 
 sense to make a standard HTTP library.

BTW, the actual JSON module in viba.data.json in richer in functionality and easier to use than the one in std.json. The module has virtually no dependency on other parts of vibe.d. I whish it was also moved to Phobos or at least inspired a refactoring of std.json. Nicolas
Nov 20 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 20 November 2012 at 04:26:02 UTC, Andrei Alexandrescu 
wrote:
 We sorely need a server expert on board with the time and 
 inclination to write a good server-side networking framework.

I agree with this but it really should be generic; working at probably the file descriptor level, so we can do whatever protocols along with files, pipes, ... and ideally, any kind of injected event. I'd really like it if we could run one event loop and be able to do keyboard/joystick input, timers, GUI, threads, etc., including third party libraries, as well as networking. (Combining two different event loops is a pain in the butt.) I have an implementation in mind (I'd use select() because I don't know libev, but we should be able to change that without changing the interface)... ... but I'm behind on my other work and have yet another deadline looming so can't do it myself right now. I can't wait till I'm retired!
 And they milk that for all it's worth: any discussion, article, 
 or blog post about Go gravitates toward the five-lines HTTP 
 server with the same implacable reach as conversations with 
 ideologists, which inevitably converge towards their 
 ideological stronghold.

You know, I'll never understand this. Anybody can do a five line http server. Hell, with my lib: import arsd.cgi; void hello(Cgi cgi) { cgi.write("hello!"); } mixin GenericMain!hello; Look, tiny http server (on port 8080)! Also works as cgi, fastcgi, scgi, and easy command line testing with a different compile option. But not really that special language wise... yay, you can call a library function. Who can't do that?
Nov 20 2012
prev sibling next sibling parent "Pragma Tix" <bizprac orange.fr> writes:
On Tuesday, 20 November 2012 at 12:12:27 UTC, Nicolas Sicard 
wrote:
 On Tuesday, 20 November 2012 at 03:49:57 UTC, Tyler Jameson 
 Little wrote:
 Would a minor refactor of vibe.d be acceptable? This is pretty 
 much what I'm looking for:

 https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d
 https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d

 Except that I'd remove some parts of it, like JSON parsing.

 I think that vibe.d could benefit from moving some of the code 
 there into Phobos. I guess it comes down to whether it makes 
 sense to make a standard HTTP library.

BTW, the actual JSON module in viba.data.json in richer in functionality and easier to use than the one in std.json. The module has virtually no dependency on other parts of vibe.d. I whish it was also moved to Phobos or at least inspired a refactoring of std.json. Nicolas

vibe.d is MIT licensed. (Like almost every other WEB related library) Guess you have to ask Soenke for a permission.
Nov 20 2012
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 20, 2012 at 03:09:23PM +0100, Adam D. Ruppe wrote:
 On Tuesday, 20 November 2012 at 04:26:02 UTC, Andrei Alexandrescu
 wrote:
We sorely need a server expert on board with the time and
inclination to write a good server-side networking framework.

I agree with this but it really should be generic; working at probably the file descriptor level, so we can do whatever protocols along with files, pipes, ... and ideally, any kind of injected event. I'd really like it if we could run one event loop and be able to do keyboard/joystick input, timers, GUI, threads, etc., including third party libraries, as well as networking. (Combining two different event loops is a pain in the butt.)

The event loop itself should be a generic component in Phobos, not specific to anything else. It should work at the file descriptor level, perhaps even combining with signal handling (using the self-pipe trick), and handle timers and scheduled events as well. Everything else should be pluggable into this generic module. I have written such things before in C/C++, but in D this will be much easier to implement, thanks to delegates (which make writing event-loop based programs so much less painful). Having a standard event loop will make it much less painful to integrate many components together in a seamless way. Combining two event loops, as Adam says, is a royal pain in the neck. Making the generic event loop a Phobos standard will eliminate this problem.
 I have an implementation in mind (I'd use select() because I don't
 know libev, but we should be able to change that without changing
 the interface)...

The interface should be generic enough that you don't have to worry about whether select or poll (or whatever the windows equivalent is) is used. I think it's a good idea to implement the select version first, as that's generic across Posixes, whereas libev is Linux-specific IIRC. T -- 2+2=4. 2*2=4. 2^2=4. Therefore, +, *, and ^ are the same operation.
Nov 20 2012
prev sibling next sibling parent "jerro" <a a.com> writes:
 whereas libev is Linux-specific IIRC.

Libev is a polling abstraction. It has select, epoll and kqueue backends (and there may be others I don't know about). So it is particulary useful for unices, where epoll (linux) and kqueue (OSX and BSD) are preferred async IO mechanisms. It can also be used on Windows, but with the select() backend, which doesn't scale well with many file descriptors. select() is also limited to 64 file descriptors by default on windows IIRC (that limit can be raised, but only to 4096 if I am not mistaken).
Nov 20 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 20 November 2012 at 15:21:12 UTC, H. S. Teoh wrote:
 It should work at the file descriptor level, perhaps even
 combining with signal handling (using the self-pipe trick)

I was thinking we'd send pointers and a type hash through a pipe. It'd keep the read/write very simple. Now, the event data must be defined to be transient, unless specifically told otherwise (e.g. immutable types), because then we can reuse the data structures to avoid allocations. But take a gander at this: http://arsdnet.net/dcode/typehash.d We can take any type tuple, do a mangle of and get a unique name for any combination of arguments. Then hash it, and we have a nicely unique, fixed size message to send down the pipe. One one end, we'll cast to a void* and send that along with the hash. On the other side, only listener functions who's ParameterTypeTuple matches the typehash will receive a message. They can then cast the void* back to the argument type with confidence. This lets us define custom messages as plain types: struct MyMessage { string text; } event.listen((MyMessage m) { writeln("Received: ", m.text); } event.send(MyMessage("hello!")); event.loop(); // we should receive the message and call the above func If there's no listener that matches the hash, we ignore the message. Now there'd be some care needed to not send pointers to stack data down the pipe, as that could be destroyed, but eh. Some details gotta be worked out but I really like the idea of using type names like this.
Nov 20 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 20 November 2012 at 16:09:37 UTC, Adam D. Ruppe wrote:
 But take a gander at this:
 http://arsdnet.net/dcode/typehash.d

I have such a hard time focusing on work when there's something like this to play with.... updated the link to show the concept actually working, not just hashing. All event listeners must take exactly one argument and it is all statically typed, but it works and can do the hash+pointer message. The hash function down there is borrowed from druntime. I couldn't find it accessible as a stdlib import so I copy/pasted. But ignoring that, the rest is actually simple code. Problem is this probably isn't safe... and might not be trusted either because you can send stack info.. maybe we could do further constraints, or copy the data into a queue, or something. I really think we could make it work though and it'd be pretty cool.
Nov 20 2012
prev sibling parent "Rob T" <rob ucora.com> writes:
On Tuesday, 20 November 2012 at 12:12:27 UTC, Nicolas Sicard 
wrote:
 BTW, the actual JSON module in viba.data.json in richer in 
 functionality and easier to use than the one in std.json. The 
 module has virtually no dependency on other parts of vibe.d. I 
 whish it was also moved to Phobos or at least inspired a 
 refactoring of std.json.

 Nicolas

I have not yet looked at the vibe.d JSON yet, but I certainly will. I agree with you that the std.json could use some improvement, it's not all that fun to use that's for sure. --rt
Nov 20 2012