www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DConf 2013 Day 2 Talk 4: Web Development in D by Vladimir Panteleev

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On reddit:

http://www.reddit.com/r/programming/comments/1fkr5s/dconf_2013_day_2_talk_4_web_development_in_d_by/

On hackernews:

https://news.ycombinator.com/item?id=5812723

On facebook:

https://www.facebook.com/dlang.org/posts/650767074936977

On twitter:

https://twitter.com/D_Programming/status/341527862815367168


Andrei
Jun 03 2013
next sibling parent "Mr. Anonymous" <mailnew4ster gmail.com> writes:
On Monday, 3 June 2013 at 12:14:48 UTC, Andrei Alexandrescu wrote:

 On reddit:

 http://www.reddit.com/r/programming/comments/1fkr5s/dconf_2013_day_2_talk_4_web_development_in_d_by/

 On hackernews:

 https://news.ycombinator.com/item?id=5812723

 On facebook:

 https://www.facebook.com/dlang.org/posts/650767074936977

 On twitter:

 https://twitter.com/D_Programming/status/341527862815367168


 Andrei


http://dconf.org/2013/talks/panteleev.pdf
404 error
Jun 03 2013
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 http://www.reddit.com/r/programming/comments/1fkr5s/dconf_2013_day_2_talk_4_web_development_in_d_by/


On Reddit they seem to suggest the idea of good stack traces for 
fibers...

Bye,
bearophile
Jun 03 2013
prev sibling next sibling parent reply "Diggory" <diggsey googlemail.com> writes:
On Monday, 3 June 2013 at 12:14:48 UTC, Andrei Alexandrescu wrote:

 On reddit:

 http://www.reddit.com/r/programming/comments/1fkr5s/dconf_2013_day_2_talk_4_web_development_in_d_by/

 On hackernews:

 https://news.ycombinator.com/item?id=5812723

 On facebook:

 https://www.facebook.com/dlang.org/posts/650767074936977

 On twitter:

 https://twitter.com/D_Programming/status/341527862815367168


 Andrei


Great talk! Would love to see the improvements to phobos 
suggested.

An idea for the virtual address space problem on 32-bit:
- Assuming each stack has a marker page at the end to prevent 
overflow, by simply exchanging the stack memory with a separate 
block of memory you can reduce the number of marker pages saving 
up to one page per fiber
- Could use some fast compression method to save a not recently 
used stack in memory
- As a last resort can save stack to a file and read it in again 
when it is required. Since events are queued the event loop can 
easily peek ahead in the queue and start loading in a stack early 
so that it is ready by the time that event gets to the front.
Jun 03 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 3 June 2013 at 18:07:57 UTC, Diggory wrote:

 Great talk! Would love to see the improvements to phobos 
 suggested.

 An idea for the virtual address space problem on 32-bit:
 - Assuming each stack has a marker page at the end to prevent 
 overflow, by simply exchanging the stack memory with a separate 
 block of memory you can reduce the number of marker pages 
 saving up to one page per fiber
 - Could use some fast compression method to save a not recently 
 used stack in memory
 - As a last resort can save stack to a file and read it in 
 again when it is required. Since events are queued the event 
 loop can easily peek ahead in the queue and start loading in a 
 stack early so that it is ready by the time that event gets to 
 the front.


Thanks!

One of the main advantages of fibers over threads is the low 
overhead for switching - basically you save registers and change 
ESP. By comparison, switching to another thread requires an OS 
kernel system call. Copying entire stacks might negate this 
performance benefit. It's worth noting that it looks like the 
average stack size will grow for D programs in the future, as the 
push is made to minimize heap allocations throughout Phobos and 
use the stack more.
Jun 03 2013
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Monday, 3 June 2013 at 18:51:45 UTC, Vladimir Panteleev wrote:

 An idea for the virtual address space problem on 32-bit:




[snip]

Another problem is passing objects on the stack across fibers. 
This might even be implicit (e.g. due to delegates).
Jun 03 2013
prev sibling parent "Diggory" <diggsey googlemail.com> writes:
On Monday, 3 June 2013 at 18:51:45 UTC, Vladimir Panteleev wrote:

 On Monday, 3 June 2013 at 18:07:57 UTC, Diggory wrote:

 Great talk! Would love to see the improvements to phobos 
 suggested.

 An idea for the virtual address space problem on 32-bit:
 - Assuming each stack has a marker page at the end to prevent 
 overflow, by simply exchanging the stack memory with a 
 separate block of memory you can reduce the number of marker 
 pages saving up to one page per fiber
 - Could use some fast compression method to save a not 
 recently used stack in memory
 - As a last resort can save stack to a file and read it in 
 again when it is required. Since events are queued the event 
 loop can easily peek ahead in the queue and start loading in a 
 stack early so that it is ready by the time that event gets to 
 the front.


 Thanks!

 One of the main advantages of fibers over threads is the low 
 overhead for switching - basically you save registers and 
 change ESP. By comparison, switching to another thread requires 
 an OS kernel system call. Copying entire stacks might negate 
 this performance benefit. It's worth noting that it looks like 
 the average stack size will grow for D programs in the future, 
 as the push is made to minimize heap allocations throughout 
 Phobos and use the stack more.


Yes, although it would theoretically be possible to swap the 
stack by swapping the mappings rather than the memory itself, 
although I doubt many OSes would support that kind of 
functionality...

I guess it's not too much to ask to use a 64-bit OS for when 
1000s of connections need to be handled!
Jun 03 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
03-Jun-2013 22:07, Diggory пишет:

 On Monday, 3 June 2013 at 12:14:48 UTC, Andrei Alexandrescu wrote:

 On reddit:

 http://www.reddit.com/r/programming/comments/1fkr5s/dconf_2013_day_2_talk_4_web_development_in_d_by/


 On hackernews:

 https://news.ycombinator.com/item?id=5812723

 On facebook:

 https://www.facebook.com/dlang.org/posts/650767074936977

 On twitter:

 https://twitter.com/D_Programming/status/341527862815367168


 Andrei


 Great talk! Would love to see the improvements to phobos suggested.


Indeed.


 An idea for the virtual address space problem on 32-bit:
 - Assuming each stack has a marker page at the end to prevent overflow,
 by simply exchanging the stack memory with a separate block of memory
 you can reduce the number of marker pages saving up to one page per fiber
 - Could use some fast compression method to save a not recently used
 stack in memory
 - As a last resort can save stack to a file and read it in again when it
 is required. Since events are queued the event loop can easily peek
 ahead in the queue and start loading in a stack early so that it is
 ready by the time that event gets to the front.



Copying to disk is certainly strange and rising the cost of context 
switch by on-the-fly compression is even more so. So is copying memory.

Since we know there is plenty RAM but limited address space we can go 
for MM file and have some say 512M of it mapped at any given time (to 
have multiple smaller windows). Think of memory window as a slot for 
fiber - i.e. any fiber is mapped to one of X fixed addresses. Then the 
only requirement that when it wakes up it's mapped to the same address 
it was born with. It's sort of hash-table where fixed addresses are 
slots (collision chains) and items are fiber context that got mapped there.

The amount of pages actually used would be fairly low and thus it may 
never have to pull them off the disk. In fact the moment it starts 
paging it turns into your idea of writing context to disk.

Now the question is relative latency of MapViewOfFile in this setting.
It's definitely something to measure if it's fast enough we are done 
here basically.

If not (and I guess not, at least it's a sys call) it makes sense to 
manage placement of Fibers with some kind of good strategy. Ideally ones 
that wait on the same resource (say updated index.html to read off disk) 
should wake up together and thus they better be in the same window (so 
you can map a pack of them at once).

-- 
Dmitry Olshansky
Jun 03 2013
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
vibe.d really puts cgi.d to shame when it comes to scalability. 
With 100 concurrent connections, cgi.d (especially in 
embedded_http and fastcgi modes, scgi and cgi are a little 
behind) can hold its own. Not great, I got 5000 requests per 

too bad.

In the question section though he started to talk about event 
loops. I've been thinking about that too. It isn't tied into 
cgi.d yet, but I've written a linux loop built on epoll:

https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/eventloop.d

It uses a pipe back to itself for the injection of arbitrary 
events. Works something like this:

// this helps listening on file descriptors
FileEventDispatcher dispatcher;

// args are fd, on read ready, on write ready, on error
dispatcher.addFile(0, (int fd) {
     ubyte[100] buffer;
     auto got = unix.read(fd, buffer.ptr, buffer.length);
     if(got == -1)
          throw new Exception("wtf");
     if(got == 0)
        exit;
     else
         writeln(fd, " sent ", cast(string) buffer[0 .. got]);
}, null, null);

// you can also listen to events identified by type

addListener(delegate void(int a) { writeln("got ", a); });
addListener(delegate void(File a) { writeln("got ", a); });
send(20); // calls the listener above
send(stdin); // works with fancier types too

loop(); // enters the loop
}


I really like the idea of dispatching things based on type, it is 
so convenient and lets you plug anything in.

The FileEventDispatcher is different because all file descriptors 
are the same type, and we might want to differentiate it based on 
descriptor number. But you could also do addListener(FdReady) or 
something like that, i havent used this for a while.



Anyway my goal here is to prove that we can have one generic 
event loop that all libraries can use. My implementation probably 
isn't phobos bound, I'm settling for "works for me" and "proves 
it can work" but maybe the experience or some code will help when 
it is time to do a standard one.


I've plugged in my terminal.d to it (optionally, use 
-version=with_eventloop)) but that's it so far. Eventually I'll 
probably put simpledisplay and cgi.d in there as well and really 
see how it is coming together. If all goes well, we can have one 
program, one event loop, and all those various inputs handled.
Jun 03 2013
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 3 June 2013 at 19:27:02 UTC, Adam D. Ruppe wrote:

 vibe.d really puts cgi.d to shame when it comes to scalability. 
 With 100 concurrent connections [...]


I forgot to finish this thought! But you read that right, at puny 
100 concurrent connections it holds its own, but if you go up to 
1000 or 5000 like the vibe.d benchmarks, my code starts dropping 
connections fast.

My view has always been "I'll cross that bridge when I come to 
it" though. The sites I work on do ok, but they don't have that 
kind of traffic!
Jun 03 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
Torrents and links: http://semitwist.com/download/misc/dconf2013/
Jun 03 2013
prev sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
Referring to the last question: Hibernate-D is *not* based on Vibe.d.
But I have already been looking into the idea of using Hibernate-D and
Vibe.d together. In fact, my recent commits to mysql-native adding
support for Phobos sockets was a big part of that.

The main issue is that Vibe.d programs should be using Vibe.d
sockets instead of ordinary sockets (I don't know what would happen if
you don't use Vibe.d sockets. My guess is it just wouldn't happen
asynchronously, but Sonke could answer that better). But Hibernate-D
aims to be usable even without Vibe.d, so it uses Phobos sockets for
MySQL (via a modified fork of an older mysql-native), and for PostgreSQL
and SQLite it just uses the C libs.

I've already converted mysql-native to support both Vibe.d and Phobos
sockets, and I've already created a branch of Hibernate-D that makes
Hibernate-D use the new official mysql-native and therefore
automatically switch to Vibe.d sockets whenever Vibe.d is being used
(detected by -version=Have_vibe_d, which is automatically added by DUB
is you're using both Vibe.d and DUB). But G^DF*^ CKD*#MMIT I *just now*
noticed that commit (and the pull request I could have sworn I made)
seems to have completely disappeared without a trace...Shit, I gotta
figure out what happened and where the hell it went...

Ugh, anyway, I've been digging through Hibernate-D's source the last
couple days checking out what else might be needed. As long as I get my
magical disappearing commit resurrected, it *looks* to me like the only
other thing that might be needed is to bypass Hibernate-D's built-in
connection pool. Even that might still work as-is (I haven't tried),
but it's not really necessary for Vibe.d users since Vibe.d has its
own fiber-safe connection pool system. But it's pretty easy to
bypass Hibernate-D's connection pool in favor of Vibe.D's
connection pool in user-code without even patching Hibernate-D.

AFAICT so far, it looks like everything else in Hibernate-D should work
fine with Vibe.d.

tl;dr:

Hibernate-D does not use Vibe.d, but I have personal interest
in using them together and I've been checking into it. Not sure what
can be done about PostgreSQL and SQLite (I *think* they'll work but
just not asynchronously - not sure what else can/should be done, I'd
have to ask Sonke). But for MySQL, all you *should* need is a patch or
two that I've been working on.
Jun 03 2013
next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Mon, 3 Jun 2013 18:11:37 -0400
Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> wrote:


 Referring to the last question: Hibernate-D is *not* based on Vibe.d.
 But I have already been looking into the idea of using Hibernate-D and
 Vibe.d together. In fact, my recent commits to mysql-native adding
 support for Phobos sockets was a big part of that.
 
 The main issue is that Vibe.d programs should be using Vibe.d
 sockets instead of ordinary sockets (I don't know what would happen if
 you don't use Vibe.d sockets. My guess is it just wouldn't happen
 asynchronously, but Sonke could answer that better). But Hibernate-D
 aims to be usable even without Vibe.d, so it uses Phobos sockets for
 MySQL (via a modified fork of an older mysql-native), and for
 PostgreSQL and SQLite it just uses the C libs.
 
 I've already converted mysql-native to support both Vibe.d and Phobos
 sockets, and I've already created a branch of Hibernate-D that makes
 Hibernate-D use the new official mysql-native and therefore
 automatically switch to Vibe.d sockets whenever Vibe.d is being used
 (detected by -version=Have_vibe_d, which is automatically added by DUB
 is you're using both Vibe.d and DUB). But G^DF*^ CKD*#MMIT I *just
 now* noticed that commit (and the pull request I could have sworn I
 made) seems to have completely disappeared without a trace...Shit, I
 gotta figure out what happened and where the hell it went...
 
 Ugh, anyway, I've been digging through Hibernate-D's source the last
 couple days checking out what else might be needed. As long as I get
 my magical disappearing commit resurrected, it *looks* to me like the
 only other thing that might be needed is to bypass Hibernate-D's
 built-in connection pool. Even that might still work as-is (I haven't
 tried), but it's not really necessary for Vibe.d users since Vibe.d
 has its own fiber-safe connection pool system. But it's pretty easy to
 bypass Hibernate-D's connection pool in favor of Vibe.D's
 connection pool in user-code without even patching Hibernate-D.
 
 AFAICT so far, it looks like everything else in Hibernate-D should
 work fine with Vibe.d.
 
 tl;dr:
 
 Hibernate-D does not use Vibe.d, but I have personal interest
 in using them together and I've been checking into it. Not sure what
 can be done about PostgreSQL and SQLite (I *think* they'll work but
 just not asynchronously - not sure what else can/should be done, I'd
 have to ask Sonke). But for MySQL, all you *should* need is a patch or
 two that I've been working on.
 


Oh wait, I totally forgot:

DDBC was moved out of HibernateD into a separate project, and *that's*
where the "missing" commit was. I was looking for it in the wrong
project.

Anyway, here are my branches:

https://github.com/Abscissa/hibernated/commits/misc
https://github.com/Abscissa/ddbc/commits/misc

And here's the DDBC pull request to make DDBC (the low-level DB
abstraction lib used by Hibernate-D) use the official "Vibe.d- and
Phobos-compatible" mysql-native:

https://github.com/buggins/ddbc/pull/1

So all you *should* need (for MySQL anyway) is that pull request (or my
full "misc" branches above), and then bypass Hibernate-D's
connection pool in favor of Vibe.d's by changing this part of your code
from:

SessionFactory factory = new SessionFactoryImpl(mySchema, myDialect,
myDataSource);

to something like:

class MyDataSource : DataSource {
    static mysql.db.MysqlDB vibePool;
    override Connection getConnection() {
        if(!vibePool)
            vibePool = new mysql.db.MysqlDB(/+ connection info +/);

        return vibePool.lockConnection();
    }
}

SessionFactory factory = new SessionFactoryImpl(mySchema, myDialect,
new MyDataSource());
Jun 03 2013
prev sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Monday, 3 June 2013 at 22:11:39 UTC, Nick Sabalausky wrote:

 Referring to the last question: Hibernate-D is *not* based on 
 Vibe.d.
 But I have already been looking into the idea of using 
 Hibernate-D and
 Vibe.d together. In fact, my recent commits to mysql-native 
 adding
 support for Phobos sockets was a big part of that.

 The main issue is that Vibe.d programs should be using Vibe.d
 sockets instead of ordinary sockets (I don't know what would 
 happen if
 you don't use Vibe.d sockets. My guess is it just wouldn't 
 happen
 asynchronously, but Sonke could answer that better). But 
 Hibernate-D
 aims to be usable even without Vibe.d, so it uses Phobos 
 sockets for
 MySQL (via a modified fork of an older mysql-native), and for 
 PostgreSQL
 and SQLite it just uses the C libs.

 I've already converted mysql-native to support both Vibe.d and 
 Phobos
 sockets, and I've already created a branch of Hibernate-D that 
 makes
 Hibernate-D use the new official mysql-native and therefore
 automatically switch to Vibe.d sockets whenever Vibe.d is being 
 used
 (detected by -version=Have_vibe_d, which is automatically added 
 by DUB
 is you're using both Vibe.d and DUB). But G^DF*^ CKD*#MMIT I 
 *just now*
 noticed that commit (and the pull request I could have sworn I 
 made)
 seems to have completely disappeared without a trace...Shit, I 
 gotta
 figure out what happened and where the hell it went...

 Ugh, anyway, I've been digging through Hibernate-D's source the 
 last
 couple days checking out what else might be needed. As long as 
 I get my
 magical disappearing commit resurrected, it *looks* to me like 
 the only
 other thing that might be needed is to bypass Hibernate-D's 
 built-in
 connection pool. Even that might still work as-is (I haven't 
 tried),
 but it's not really necessary for Vibe.d users since Vibe.d has 
 its
 own fiber-safe connection pool system. But it's pretty easy to
 bypass Hibernate-D's connection pool in favor of Vibe.D's
 connection pool in user-code without even patching Hibernate-D.

 AFAICT so far, it looks like everything else in Hibernate-D 
 should work
 fine with Vibe.d.

 tl;dr:

 Hibernate-D does not use Vibe.d, but I have personal interest
 in using them together and I've been checking into it. Not sure 
 what
 can be done about PostgreSQL and SQLite (I *think* they'll work 
 but
 just not asynchronously - not sure what else can/should be 
 done, I'd
 have to ask Sonke). But for MySQL, all you *should* need is a 
 patch or
 two that I've been working on.


Regarding PostgreSQL, keep in mind that it has an async API, 
which would be handy in building a Vibe-friendly wrapper:

http://www.postgresql.org/docs/devel/static/libpq-async.html

Graham
Jun 03 2013