www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - web development in D

reply joe <joe studiofortress.com> writes:
I currently do most of my web development in PHP, with some work in Ruby with
RoR. Right now I'm starting to think about building my own stack for web dev
(I already use my own MVC framework and libs in PHP), but I'd really like to
move to something faster and more powerful. Java or ASP.NET are two obvious
choices, but I'm currently researching other alternatives (like D).

So first I was wondering if there has been any web development using D, and if
there are many (or any) libraries/frameworks/other resources already available
for this?

Second, from your own experience using D do think it would make a good or bad
choice for a web development language?
May 21 2011
next sibling parent Trass3r <un known.com> writes:
Have a look at the recent thread titled 'How To Dynamic Web Rendering?'
Adam Ruppe has created a package for web development with D and it seems  
to work like a charm.
May 21 2011
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Yea, the Dynamic Web Rendering thread just went over this, so
shouldn't be hard to find that thread.

It went into discussing my method and got a little long, but the
summary is:

You can write web apps in D using the standard CGI interface or
a long lived process (embedded http server or whatever.)

My code works best with standard CGI, cgi.d in here:
http://arsdnet.net/dcode/

I also have an httpd.d (requires netman.d) that lets the cgi
interface work with an embedded server, but it's single threaded
so that can be a problem. It's not production ready.

Regular cgi though works with other existing web servers, is
easy to use, and actually performs quite well.

(You'll find a lot of people complaining on the internet that
CGI is slow, but check the date on those articles and see if they
are using Perl or native compiled binaries.

Usually they are very old and using an interpreter - that's why
it is slow. A native binary over cgi is quite fast.)



The stuff you build on top of cgi can be whatever you want. I
personally use my dom.d and web.d modules for templating and
some automatic code generation, but you can spin your own libs
too.
May 21 2011
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 21 May 2011 19:18:39 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 (You'll find a lot of people complaining on the internet that
 CGI is slow, but check the date on those articles and see if they
 are using Perl or native compiled binaries.

 Usually they are very old and using an interpreter - that's why
 it is slow. A native binary over cgi is quite fast.)

Regular CGI still has the overhead of starting a new process for every request. PHP without CGI is likely to be faster than D with CGI. OP is looking for best performance, so CGI is not a good recommendation - FastCGI/webserver module/built-in webserver are the way to go. I've had great experience with using an HTML/jQuery frontend with a D backend (built-in HTTP server which only serves static files and AJAX requests). -- Best regards, Vladimir mailto:vladimir thecybershadow.net
May 22 2011
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:
 Regular CGI still has the overhead of starting a new process for
 every request.

That's a very small cost - between 1 and 5 milliseconds, depending on your hardware. Next to the network latency involved in making a request, that's literally nothing - it's less than random network variation, so it has no impact on the end user experience. It also disappears quickly if your program does any processing, since it's like a line: y = mx + b. The startup cost is b, and the speed of the language is m. If x is even slightly big, reducing m means you'll get better end results than reducing b. (unless b is *huge*... like network latency. Adding an "Expires:" header to your responses gives the biggest gains of all.) My work application runs about 40% faster on CGI than the fastest comparable product we've found in mod_php, on average, with certain CPU intensive portions being up to 5x faster. That said, an embedded webserver /is/ even faster, but it's also harder to use. The end result depends on your app. Only actually benchmarking it can tell for sure what matters. My cgi.d provides the same interface regardless of how it is called - switching to built in httpd is easy if needed.
 I've had great experience with using an HTML/jQuery frontend with
 a D backend

From what I've seen in my benchmarks, you would probably get a bigger usability boost by ditching jQuery than by ditching CGI. Even when cached, jQuery takes a while for the browser to parse - comparable to starting up a native process on the server.
May 22 2011
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
 I've never done any CGI stuff before, but can't you employ some kind
 of messaging mechanism with an already running process?

Yes. I find it's quite useful when you want instant notifications on something, like a web chat application, but it could do some speed boosts too. I'm working on a post showing some of my benchmarks too on built-in vs cgi. Some of them are over 20x faster with a built in webserver! (I wonder if this is cgi's fault or if Apache is just not optimized for it... I'm changing the i/o code too so we can make sure it's not just stdio being slow.)
May 22 2011
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 22/05/2011 19:11, Adam D. Ruppe wrote:
 I've never done any CGI stuff before, but can't you employ some kind
 of messaging mechanism with an already running process?

Yes. I find it's quite useful when you want instant notifications on something, like a web chat application, but it could do some speed boosts too. I'm working on a post showing some of my benchmarks too on built-in vs cgi. Some of them are over 20x faster with a built in webserver! (I wonder if this is cgi's fault or if Apache is just not optimized for it... I'm changing the i/o code too so we can make sure it's not just stdio being slow.)

Have you tried FastCGI? It's essentially CGI but with a long running process rather than re-spawning for each request (and it still has the advantage that it can be re-spawned by the web server). I'm using it for serenity, albeit the FCGX interface rather than the interface that emulates CGI. -- Robert http://octarineparrot.com/
May 22 2011
parent Adam D. Ruppe <destructionator gmail.com> writes:
Robert Clipsham wrote:
 Have you tried FastCGI?

I haven't. I read about it but concluded it would actually be just about as difficult to implement as a minimalist http server, so I never went into it. (http itself is hard to get all the corner cases right, but if you live behind something like Apache, you can get by without a full implementation. But in both cases, you have to write a fast network layer, and that's the harder part.) Besides though, there's a penalty between D in CGI and D with long lived processes.... but D in CGI beats PHP a lot of the time, so it still wins. I wonder how D/cgi stacks up against a long lived Python or Ruby process. I doubt they'd do better than embedded PHP, but I haven't tried them so I'm not sure.
May 22 2011
prev sibling next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
I wrote:
 That said, an embedded webserver /is/ even faster

I measured this using one of my in-development apps recompiled to use the built in server I have, picking a page with moderate processing requirements. CGI: Time per request: 17.375 [ms] (mean) built in webserver: Time per request: 8.727 [ms] (mean) I wonder why the difference is bigger than the startup I measured when isolated? I think I/O contributes to slowdown a little too. My startup cost was based on hello world, no significant output. This outputs several kilobytes of data. That's without any optimization. Let's try that again, making the mysql and reflection objects static so they needn't be recreated on every request when embedded.... Cgi: Time per request: 17.178 [ms] (mean) Built-in: Time per request: 5.079 [ms] (mean) Very nice. Those are all pretty small numbers, but we do see a clear speedup with the built in webserver. Since my webserver is single threaded, I don't think this is really fit for production use. If one user's request went too long - which it shouldn't, but could - it'd slow everyone down. It also can't serve up staticish files while waiting on something else to process. Another worry I have with a long lived process is what if it dies? With CGI, you can trust Apache to start a fresh copy for you. That's why I call the cgi one easier to use. You just drop it in there, no need to worry about it dying, no need to start or restart the process for updates, etc. For fun, let's run it on some simpler pages too Serving the auto-generated Javascript api file: CGI: Time per request: 13.678 [ms] (mean) Built in: Time per request: 1.070 [ms] (mean) Big multiplier, but since it's a trivial program, we can see it's about the same difference as we saw above - a constant ~10-13 ms savings. I also serve the stylesheet through it, because then it's deployed all as one file. This is done before the database and reflection objects are created anyway, so that optimization shouldn't have an effect here. CGI: Time per request: 10.493 [ms] (mean) Built in: Time per request: 0.774 [ms] (mean) And indeed, it didn't - the 10 ms constant cost is all we see. I wonder if I can do anything to improve the I/O performance.... nope, changing to the syscall from stdio made no significant difference. Doing time PATH_INFO=/style.css ./cgi-app > /dev/null averages to ~10 ms too, so I can't blame Apache either. So startup + object construction + I/O overhead for a real world page gives CGI about a 10ms penalty. This ranges from a small to a very big multiplier of total time, depending on what the program is. Next to real world network conditions, this is consistently a small number in the big picture - the end user is unlikely to notice it.
May 22 2011
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:
 But if your website is getting enough hits to generate more
 requests than the server can process, technology choice matters a
 lot.

Yeah. I've never had that happen, so I don't really know. If it happens, it's easy enough to change later. (it was a two line change just now to switch it to built-in webserver - none of the actual app code needs to change at all)
 Is there any reason you didn't go for FastCGI or SCGI?

The biggest reason is they are harder to implement, and it's premature optimization. I didn't want to spend a lot of extra time writing code that would never be needed! Secondary benefits are simplicity and reliability. A CGI process can segfault without affecting anyone else. It can go completely wild on memory, and it doesn't matter because it's short-lived anyway. It can deployed to any server with ease - just copy the binary in there. Worst case, you add a couple lines to .htaccess. Apache (or IIS) also handles logging and other details for a CGI program. It's just simpler in a lot of ways.
May 22 2011
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 22/05/2011 22:40, Adam D. Ruppe wrote:
 Vladimir Panteleev wrote:
 But if your website is getting enough hits to generate more
 requests than the server can process, technology choice matters a
 lot.

Yeah. I've never had that happen, so I don't really know. If it happens, it's easy enough to change later. (it was a two line change just now to switch it to built-in webserver - none of the actual app code needs to change at all)
 Is there any reason you didn't go for FastCGI or SCGI?

The biggest reason is they are harder to implement, and it's premature optimization. I didn't want to spend a lot of extra time writing code that would never be needed!

FastCGI has an interface available that emulates CGI - that's not exactly harder to implement! ;)
 Secondary benefits are simplicity and reliability. A CGI process
 can segfault without affecting anyone else. It can go completely
 wild on memory, and it doesn't matter because it's short-lived
 anyway. It can deployed to any server with ease - just copy
 the binary in there. Worst case, you add a couple lines to
 .htaccess.

Same for FCGI, for the most part, except it's a long running process. You can set it up to periodically re-spawn if you like, or it will automatically re-spawn when you exit (error or otherwise).
 Apache (or IIS) also handles logging and other details for a CGI
 program.

Same for FCGI.
 It's just simpler in a lot of ways.

That's debatable! :D -- Robert http://octarineparrot.com/
May 22 2011
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
Robert Clipsham wrote:
 FastCGI has an interface available that emulates CGI - that's not
 exactly harder to implement

What scared me was that the data comes in on a socket... so wouldn't that come with the same threading problems my simple http server has? (Where one slow request means everyone else has to wait) I guess I'll have to try it! [a little later] So I changed cgi.d to include support for fast cgi by linking in one of the C libraries. That wasn't hard. It was kinda painful to get the C library and apache module working though... Anyway, I recompiled my same in-development work application to use fcgi, so let's run the benchmarks on it and see what we got. Same page as I used in the last post. 7 ms per request under regular cgi it ran in about 20 ms. My embedded httpd served it up in 5 ms per request. Turning up the concurrent requests to 50 (I'm using the program ab for these benchmarks btw) shows me that Apache spawned more copies of the process automatically. Excellent. Got 150 ms with fcgi. 35 ms with embedded. 370 ms with plain cgi. (I wonder why the plain cgi shows weirdness here) With 100, my embedded one still wins by a lot in 90% of requests, but in the other ones, it gets brutally slow - 600 ms for 5% and 9 full seconds for 1% of them. Ouch! fcgi came in at 250 for most requests, with the longest one being a little over a second. Getting the generated Javascript file: -c 50, CGI: 300 ms -c 50, FCGI: 50 ms (nice!) -c 50, embedded: 6 ms for 95%, but one took 3 seconds. Getting the static stylesheet, still -c 50: CGI: 150 ms FCGI: 50 ms embedded: 4ms -c 1 CGI: 12 ms FCGI: 1 ms embedded: 1 ms Keep in mind, this program was *already* beating comparable mod_php applications, even in plain CGI. Now, it's just a curbstomp. Overall, that's pretty ok, there is a consistent boost over plain CGI. My simple embedded server seems to do consistently better though, until it hits the concurrency wall. (I'd be willing to bet a big part of the reason is that the embedded one uses slices for a lot of stuff. The other ones copy data at least once.) Not bad at all, but I'm not sure it's good enough for me to switch to it in general. Regardless, this proves something I've been saying though: my cgi.d adapts with ease to different environments. I'm running the same program in these benchmarks. Only difference is compiling with -version=fastcgi or -version=embedded_httpd. The app code is completely unchanged.
May 22 2011
parent Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:
 Have you heard about NodeJS?

Yeah. It's actually not too much different than the way my network manager code works in D (which the embedded http server is made on) although their implementation is much better than mine - I just use select() in the event loop. The key to using it is to always return from your function as soon as possible. It will then add to your buffer and call the function again as new stuff comes in. It actually works quite well for it's simplicity. The problem with using it for the cgi apps is they aren't really written to work in that style.
May 22 2011
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
Vladimir Panteleev wrote:
 PHP without CGI is likely to be faster than D with CGI.

Nope. In my last post, I did cgi/D vs builtin/D and we saw a 10 ms difference (running on my home computer. I did the same thing on the live company server just now and saw only a 2 ms difference - proper configuration and hardware helps cut it down too.) I mentioned a 40% speedup on comparable products D vs PHP. Let's list some more. One of the company websites is run by Wordpress. 246 ms per request. Another uses phpBB3. 73 ms per request. One of them uses a PHP graph library to make a quick line graph. 136 ms per request. Let's contrast to an assortment of D sites I've written for them, all on that same server. Dynamic homepage: 11 ms per request Scheduling and conferencing app: 14 ms per request Gradient generator outputting small pngs: 4 ms per request Most the D apps would be too hard to rewrite in PHP to do a direct comparison, so we'll have to settle for the 10x difference we see between these and the popular PHP packages... But the gradient generator is simple. I found a PHP program on the web that does the same thing with basic options, utilizing the GD library. I downloaded and benchmarked it.... 19 ms per request, creating identical images. (average of 2000 requests) Recall the D program, running on CGI, did the very same thing in 4 ms per request, about 5x faster. We can ignore the network, embrace CGI, and still *easily* crush PHP performance with anything more complicated than hello world.
May 22 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I've never done any CGI stuff before, but can't you employ some kind
of messaging mechanism with an already running process? Maybe use
memory-mapped files or some other technique to communicate with the
process?
May 22 2011
prev sibling next sibling parent Trass3r <un known.com> writes:
Very interesting benchmarks!
May 22 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 22 May 2011 20:10:07 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 Vladimir Panteleev wrote:
 Regular CGI still has the overhead of starting a new process for
 every request.

That's a very small cost - between 1 and 5 milliseconds, depending on your hardware. Next to the network latency involved in making a request, that's literally nothing - it's less than random network variation, so it has no impact on the end user experience.

Yes, of course - the difference is negligible if the server load is low. But if your website is getting enough hits to generate more requests than the server can process, technology choice matters a lot.
 It also disappears quickly if your program does any processing,
 since it's like a line: y = mx + b. The startup cost is b, and
 the speed of the language is m.

Yes, I should have added that I meant for very simple requests. The speed advantage of a compiled language will always overtake most initialization overheads for non-trivial requests.
 The end result depends on your app. Only actually benchmarking it
 can tell for sure what matters. My cgi.d provides the same interface
 regardless of how it is called - switching to built in httpd is
 easy if needed.

Is there any reason you didn't go for FastCGI or SCGI? The main advantage I can see is that CGI is much simpler for edit-compile-test iterations.
 I've had great experience with using an HTML/jQuery frontend with
 a D backend

From what I've seen in my benchmarks, you would probably get a bigger usability boost by ditching jQuery than by ditching CGI. Even when cached, jQuery takes a while for the browser to parse - comparable to starting up a native process on the server.

The difference here is that HTML, CSS and jQuery will only be loaded once. I was talking about AJAX web-apps. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
May 22 2011
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 22 May 2011 22:41:12 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 We can ignore the network, embrace CGI, and still *easily* crush
 PHP performance with anything more complicated than hello world.

Interesting. Thanks. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
May 22 2011
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Mon, 23 May 2011 03:49:49 +0300, Adam D. Ruppe  
<destructionator gmail.com> wrote:

 What scared me was that the data comes in on a socket... so wouldn't
 that come with the same threading problems my simple http server
 has? (Where one slow request means everyone else has to wait)

Asynchronous networking is all the rage now. lighttpd is completely asynchronous and VERY fast. Have you heard about NodeJS? It uses asynchronous IO as the primary (only?) way to do IO. As closures are first-class JavaScript language features, async IO works pretty well there. I dream about creating something like this in D as well one day. Anyway, the idea is that nothing should block your event loop for more than a millisecond or so. If something takes longer, then either you need to look into "asynchronizing" it (file access?), or putting it in a separate, isolated thread. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
May 22 2011
prev sibling next sibling parent "Nick Sabalausky" <a a.a> writes:
"joe" <joe studiofortress.com> wrote in message 
news:ir8frr$ait$1 digitalmars.com...
 Second, from your own experience using D do think it would make a good or 
 bad
 choice for a web development language?

I'd really say that a good choice for a web development language is whatever you find to be a good choice for a language. Personally, I love D. It's my favorite language, and out of all the languages I've used (many over the years) it's really the only one I'm overall happy with (but then, I'm notoriously picky ;) ). So I find D to be an excellent choice for a web development language since I personally find D to be an excellect language. OTOH, if someone happened to absolutely hate D, then D probably wouldn't be a very good choice for them for a web development language. There is one other factor to consider, though: You need to think about what server or servers it will need to run on: If, for example, you need it to be able to run on pretty much any shared web host out there, then you're pretty much stuck with either PHP or something like Haxe that compiles down to PHP. And that defintiely won't help you with performance, and it'll only help alleviate *some* of PHP's probelms, not all of them. That's the unfortunate state of shared web hosting :/ If, OTOH, you only need it to run on one specific server and you don't have complete control over it (for example, if it's a shared web host, or if it's run by some separate IT department thet you're not part of), then naturally you'll need to see what the server supports and make your choice from those. But keep in mind one thing I've learned the hard way: Just because they say they support CGI does *not* necessarily mean they support CGI from a natively compiled langauge (which D is). So make sure to get clarification on that. And, of course, if you have total control over the server, then naturally you can choose whatever you want. Oh, but unless you do have direct access to the server and total control over it, I would recommend against ASP.NET. Not because I really have anything against ASP.NET, Windows or MS, per se. It's just that ASP.NET typically requires a Windows server, and shared Windows hosts usually don't provide any way to set up things like file permissions and internal URL rewriting. And not being able to do those things can often be a problem.
May 21 2011
prev sibling parent reply Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
joe wrote:

 I currently do most of my web development in PHP, with some work in Ruby
 with RoR. Right now I'm starting to think about building my own stack for
 web dev (I already use my own MVC framework and libs in PHP), but I'd
 really like to move to something faster and more powerful. Java or ASP.NET
 are two obvious choices, but I'm currently researching other alternatives
 (like D).
 
 So first I was wondering if there has been any web development using D,
 and if there are many (or any) libraries/frameworks/other resources
 already available for this?

As mentioned Adam D. Ruppe has nice libs for web development, he managed to make a living out of it too so he must be doing something right :) There is one other web framework for D2 called serenity, I'm not sure how far it is but you can check it out here: https://github.com/mrmonday/serenity Perhaps this is unnecessary to mention, but would you choose ASP.NET, I'd recommend against webforms and look into ASP.NET mvc. If haven't used the latter myself but I didn't like webforms too much and heard good things about the mvc framework. Good luck!
May 22 2011
parent Robert Clipsham <robert octarineparrot.com> writes:
On 22/05/2011 13:09, Lutger Blijdestijn wrote:
 joe wrote:

 I currently do most of my web development in PHP, with some work in Ruby
 with RoR. Right now I'm starting to think about building my own stack for
 web dev (I already use my own MVC framework and libs in PHP), but I'd
 really like to move to something faster and more powerful. Java or ASP.NET
 are two obvious choices, but I'm currently researching other alternatives
 (like D).

 So first I was wondering if there has been any web development using D,
 and if there are many (or any) libraries/frameworks/other resources
 already available for this?

As mentioned Adam D. Ruppe has nice libs for web development, he managed to make a living out of it too so he must be doing something right :) There is one other web framework for D2 called serenity, I'm not sure how far it is but you can check it out here: https://github.com/mrmonday/serenity Perhaps this is unnecessary to mention, but would you choose ASP.NET, I'd recommend against webforms and look into ASP.NET mvc. If haven't used the latter myself but I didn't like webforms too much and heard good things about the mvc framework. Good luck!

Thanks for mentioning it! :D I held off mentioning it due to its immaturity in comparison to Adam's stuff, I'm gradually working on it. The latest thing I've added is an ORM to remove about 100 lines of code for simple stuff (still a work in progress). As a result I'm probably going to be reworking the overall design (It started off being MVC, it's not going to be now). -- Robert http://octarineparrot.com/
May 22 2011