www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - zero-copy API

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
http://docs.google.com/viewer?a=v&q=cache:K15RE_6zxSwJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.134.4874%26rep%3Drep1%26type%3Dpdf+zero+copy+i/o&hl=en&gl=us&pid=bl&srcid=ADGEESjBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&sig=AHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3mJdg


Andrei
Oct 14 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:i982i6$kch$1 digitalmars.com...
 http://docs.google.com/viewer?a=v&q=cache:K15RE_6zxSwJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.134.4874%26rep%3Drep1%26type%3Dpdf+zero+copy+i/o&hl=en&gl=us&pid=bl&srcid=ADGEESjBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&sig=AHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3mJdg


 Andrei

Not to be a Debbie Downer (or "Danny Downer" in this case?), but is that available somewhere in a *real* format instead of that Google Viewer bullshit? (Ok, maybe more like "Andy Asshole" ;) )
Oct 14 2010
next sibling parent reply Jimmy Cao <jcao219 gmail.com> writes:
--90e6ba61508815726204929c4466
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Just click Download Original in the File menu.
And also, in my opinion Google Docs Viewer is an amazing piece of art.  I
use it as my default PDF opener on Chrome.

On Thu, Oct 14, 2010 at 6:33 PM, Nick Sabalausky <a a.a> wrote:

 "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message
 news:i982i6$kch$1 digitalmars.com...


zero+copy+i/o&hl=3Den&gl=3Dus&pid=3Dbl&srcid=3DADGEESjBkiUxG4hRImVjOFy886Gr= JxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQT= bljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&sig=3DAHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3m= Jdg
 Andrei

Not to be a Debbie Downer (or "Danny Downer" in this case?), but is that available somewhere in a *real* format instead of that Google Viewer bullshit? (Ok, maybe more like "Andy Asshole" ;) )

--90e6ba61508815726204929c4466 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Just click Download Original in the File menu.<div>And also, in my opinion = Google Docs Viewer is an amazing piece of art. =A0I use it as my default PD= F opener on Chrome.<br><br><div class=3D"gmail_quote">On Thu, Oct 14, 2010 = at 6:33 PM, Nick Sabalausky <span dir=3D"ltr">&lt;a a.a&gt;</span> wrote:<b= r> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex;">&quot;Andrei Alexandrescu&quot; &lt;<a href= =3D"mailto:SeeWebsiteForEmail erdani.org">SeeWebsiteForEmail erdani.org</a>= &gt; wrote in message<br> news:i982i6$kch$1 digitalmars.com...<br> <div class=3D"im">&gt; <a href=3D"http://docs.google.com/viewer?a=3Dv&;q= =3Dcache:K15RE_6zxSwJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1= .134.4874%26rep%3Drep1%26type%3Dpdf+zero+copy+i/o&amp;hl=3Den&amp;gl=3Dus&a= mp;pid=3Dbl&amp;srcid=3DADGEESjBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_ii= cbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2Lu= A2-5kKPbbEp7z6&amp;sig=3DAHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3mJdg" target=3D"_bla= nk">http://docs.google.com/viewer?a=3Dv&;q=3Dcache:K15RE_6zxSwJ:citeseer= x.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.134.4874%26rep%3Drep1%26type%= 3Dpdf+zero+copy+i/o&amp;hl=3Den&amp;gl=3Dus&amp;pid=3Dbl&amp;srcid=3DADGEES= jBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl= 1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&amp;sig=3DAHIEt= bSmuH-Y2AGdwSQxyJcbBXLRB3mJdg</a><br> &gt;<br> &gt;<br> &gt; Andrei<br> <br> </div>Not to be a Debbie Downer (or &quot;Danny Downer&quot; in this case?)= , but is that<br> available somewhere in a *real* format instead of that Google Viewer<br> bullshit? (Ok, maybe more like &quot;Andy Asshole&quot; ;) )<br> <br> <br> </blockquote></div><br></div> --90e6ba61508815726204929c4466--
Oct 14 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jimmy Cao" <jcao219 gmail.com> wrote in message 
news:mailman.613.1287099799.858.digitalmars-d puremagic.com...
Just click Download Original in the File menu.
And also, in my opinion Google Docs Viewer is an amazing piece of art.  I
use it as my default PDF opener on Chrome.

"File" menu's greyed out and don't work.
Oct 14 2010
parent reply Eric Poggel <dnewsgroup2 yage3d.net> writes:
On 10/14/2010 8:24 PM, Vladimir Panteleev wrote:
 On Fri, 15 Oct 2010 03:06:18 +0300, Nick Sabalausky <a a.a> wrote:

 "File" menu's greyed out and don't work.

Turn on JavaScript. :D *ducks*

that doesn't work without JavaScript :)
Oct 14 2010
parent reply BCS <none anon.com> writes:
Hello Eric,

 On 10/14/2010 8:24 PM, Vladimir Panteleev wrote:
 
 On Fri, 15 Oct 2010 03:06:18 +0300, Nick Sabalausky <a a.a> wrote:
 
 "File" menu's greyed out and don't work.
 

*ducks*

that doesn't work without JavaScript :)

If you did, you'd have enough money to develop something that wasn't the disaster that JS-in-the-browser turned into. -- ... <IXOYE><
Oct 17 2010
parent "Nick Sabalausky" <a a.a> writes:
"BCS" <none anon.com> wrote in message 
news:a6268ff1e8698cd3c49a2116a7c news.digitalmars.com...
 Hello Eric,

 On 10/14/2010 8:24 PM, Vladimir Panteleev wrote:

 On Fri, 15 Oct 2010 03:06:18 +0300, Nick Sabalausky <a a.a> wrote:

 "File" menu's greyed out and don't work.

*ducks*

that doesn't work without JavaScript :)


Heh :) But it's actually kinda surprising just how often I'm able to fudge my way through a site that's clearly designed for JS-only even without actually turning JS on. Rank-and-file web devs often seriously underestimate just how much can be done, and done rather easily and to good effect, without a single line of in-browser scripting like JS or Flash. Fairly recent example: A guy I've been doing some web work for sent me some page prototypes. They had graphic-link buttons with rollover effects and a data-submission form. The link rollovers were done in JS (auto-generated by Dreamweaver - yea, "ick"), and the data-submission form was done in Flash (Is there anything Adobe *can* do right?). I kinda balked at it, but he didn't see anything wrong with it. So I started from scratch, redid the rollovers in CSS2 (and you probably already know what I think of CSS ;) ), and redid the form in HTML. Took mere minutes (at least those particular parts did - the CSS layout took forever, until I decited to toss in a few <table>s). Showed them to him and all of a sudden he was thrilled with how much better both of them worked. And this wasn't another "you'll never pull my 10-year-old 32-bit box even from my cold dead hands" guy like me. Quite the opposite - this is one of those obessively upgrading "always gotta have the latest and greatest" sorts of people, so his hardware performance and software versions were way up there and he still noticed a significant improvement.
 If you did, you'd have enough money to develop something that wasn't the 
 disaster that JS-in-the-browser turned into.

Now I like that idea!
Oct 17 2010
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Fri, 15 Oct 2010 03:06:18 +0300, Nick Sabalausky <a a.a> wrote:

 "File" menu's greyed out and don't work.

Turn on JavaScript. :D *ducks* -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Oct 14 2010
prev sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 15 Oct 2010 03:08:23 +0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 http://docs.google.com/viewer?a=v&q=cache:K15RE_6zxSwJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.134.4874%26rep%3Drep1%26type%3Dpdf+zero+copy+i/o&hl=en&gl=us&pid=bl&srcid=ADGEESjBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&sig=AHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3mJdg


 Andrei

That paper is an ancient one and the API they demonstrated isn't a good one. It is a huge step backward for a language with a garbage collector to ask users to free buffers manually when they don't need them. You can't do it automatically upon next read/write operation because user can still have a pointer to it. In general, you can't reuse old buffer and it means you must allocate new one every time you issue an I/O request, and memory allocation is a more expensive operation than a memcpy. In many OS documentations (e.g. in Windows and Sony consoles that use Linux kernel) it is stated that I/O operations often use provided buffers directly (if an underlying hardware allows to), you usually don't need to do anything special for that to occur (in PlayStation Portable read requires you buffer to be aligned at 4 bytes boundary, it will fail otherwise, but when was last time you used an unaligned buffer?). IIRC, D memory allocator aligns memory at 16 bytes boundary, and it is very strange to provide an unaligned slice the read(). However, if Stream buffers anything, there can be N bytes stored in the buffer, and N can be unaligned. In this case, you have to copy these N bytes to the user-provided buffer, and then call osSpecificRead(fd, userBuffer.ptr + N, userBuffer.length - N); As a result, userBuffer.ptr + N can now be unaligned, and direct io is impossible in this case. Conclusion: Stream interface should not do any buffering, all it should do is call osSpecificRead on a user-provided buffer directly.
Oct 14 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/14/2010 07:50 PM, Denis Koroskin wrote:
 On Fri, 15 Oct 2010 03:08:23 +0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 http://docs.google.com/viewer?a=v&q=cache:K15RE_6zxSwJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.134.4874%26rep%3Drep1%26type%3Dpdf+zero+copy+i/o&hl=en&gl=us&pid=bl&srcid=ADGEESjBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&sig=AHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3mJdg



 Andrei

That paper is an ancient one and the API they demonstrated isn't a good one.

What is a better, more recent one? I noticed that libfbufs has little traction (googling for it only reveals 4 results) but the paper is quoted by 63 others, so it's fairly influential.
 It is a huge step backward for a language with a garbage collector
 to ask users to free buffers manually when they don't need them.

That's a system-level API. It can receive any amount of window dressing.
 You
 can't do it automatically upon next read/write operation because user
 can still have a pointer to it. In general, you can't reuse old buffer
 and it means you must allocate new one every time you issue an I/O
 request, and memory allocation is a more expensive operation than a memcpy.

I think that can be fixed by using sealing. Andrei
Oct 14 2010
next sibling parent Sean Kelly <sean invisibleduck.org> writes:
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 On 10/14/2010 07:50 PM, Denis Koroskin wrote:
 On Fri, 15 Oct 2010 03:08:23 +0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 
 http://docs.google.com/viewer?a=v&q=cache:K15RE_6zxSwJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.134.4874%26rep%3Drep1%26type%3Dpdf+zero+copy+i/o&hl=en&gl=us&pid=bl&srcid=ADGEESjBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&sig=AHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3mJdg
 
 
 
 Andrei

That paper is an ancient one and the API they demonstrated isn't a good one.

What is a better, more recent one? I noticed that libfbufs has little traction (googling for it only reveals 4 results) but the paper is quoted by 63 others, so it's fairly influential.

Perhaps not terribly relevant, but IOCP on Windows does this. To perform an asynchronous read you supply the buffer to recv and then an event is signaled when the read completes. The same API works for file io and there's support for scatter/gather as well. It makes for a more complicated program though.
Oct 14 2010
prev sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 15 Oct 2010 08:03:21 +0400, Sean Kelly <sean invisibleduck.org>  
wrote:

 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:
 On 10/14/2010 07:50 PM, Denis Koroskin wrote:
 On Fri, 15 Oct 2010 03:08:23 +0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 http://docs.google.com/viewer?a=v&q=cache:K15RE_6zxSwJ:citeseerx.ist.psu.edu/viewdoc/download%3Fdoi%3D10.1.1.134.4874%26rep%3Drep1%26type%3Dpdf+zero+copy+i/o&hl=en&gl=us&pid=bl&srcid=ADGEESjBkiUxG4hRImVjOFy886GrJxRuhFcePjbadiUw9h1c_iicbhhArOgd55vpk0tP6ST4KjhY1j6rl1_PN-msIExUvxSPJWuXfQTbljj4ZYyutY6wvp3mc3t2LuA2-5kKPbbEp7z6&sig=AHIEtbSmuH-Y2AGdwSQxyJcbBXLRB3mJdg



 Andrei

That paper is an ancient one and the API they demonstrated isn't a good one.

What is a better, more recent one? I noticed that libfbufs has little traction (googling for it only reveals 4 results) but the paper is quoted by 63 others, so it's fairly influential.

Perhaps not terribly relevant, but IOCP on Windows does this. To perform an asynchronous read you supply the buffer to recv and then an event is signaled when the read completes. The same API works for file io and there's support for scatter/gather as well. It makes for a more complicated program though.

We are using IOCP for a game server with a great success. I'm not even sure we could achieve the same level of latency if we were using Linux and epoll. I've implemented a Stream API (FileStream and SocketStream) on top of IOCP (both sync and async), and it doesn't complicate a program when used right. Automatic thread pooling for event callbacks is very handy, and since events for the same socket don't overlap you don't even need to synchronize anything but global state (in which case a simple ReadWriteMutex is more than enough). Unfortunately, I'm not sure how to integrate that concept into D2 design. Even though events don't overlap, the callback thread is always different and as such thread local variables are unaccessible. I was thinking about dynamic thread local variables switching, but Walter says it is either highly inefficient or impossible to to implement.
Oct 14 2010