• nazriel (74/74) Aug 11 2013 Greetings.
• Dicebot (13/21) Aug 11 2013 Such behavior may happen if you use some blocking I/O routine
• Dicebot (4/4) Aug 11 2013 Though using 1Kb buffer to read/write large file _may_ be an

"nazriel" <spam dzfl.pl> writes:

Greetings.

I am trying to learn Vibe.D and rewrite using it, my old project 
I've wrote with my (already rotting) personal framework

Project been written as commercial project for one of my clients 
so I can't really release source code here, but in tl;dr it is:

Proxy that gets file from remote server (process of getting file 
itself is rather complicated) and then serves it to the connected 
client.

In my old project I used threads because there are no more than 
15 clients connected same time. There was main thread listening 
for connections, and 15 worker threads that were making bridge 
between client and remote server.

Now I tried few different approaches with Vibe.d but all had some 
issues.
The far I could get was this:
http://dpaste.dzfl.pl/d4dabd0a [1]

But it still has some issues. It works great for small files but 
for larger
(~150MB) VibeD just closes remote connection and web browsers 
just keep trying to receive something until it timeouts.

Another issue which will probably come is the fact that Vibe.D 
won't be able to serve more than 1 client at the same time. At 
least my attempts with 2MB file resulted with: 1st client 
downloading file, 2nd client getting 500 from vibe.d

Question is then: How to properly handle it with VibeD? How do 
you handle more threads with VibeD?

Aha, and please don't get into details like "your buffer is too 
small" etc. It isn't important at this stage.

Thanks in advance!

[1]

----
import vibe.d;

import std.stdio;


void handle(HTTPServerRequest, HTTPServerResponse res)
{
     string link = "http://yourdomain.com/file_here";

     requestHTTP(link, null,
         (scope HTTPClientResponse r)
         {
             enum BUFF_SIZE = 1024;
             ubyte[BUFF_SIZE] buff;
             ulong last_read = 0;

             res.headers = r.headers;
             res.headers["Content-Disposition"] = `attachment; 
filename="fname.gz"`;

             while (r.bodyReader().empty == false)
             {
                 last_read = r.bodyReader().leastSize;
                 if (last_read > BUFF_SIZE)
                     r.bodyReader().read(buff);
                 else
                     r.bodyReader().read(buff[0..last_read]);

                 res.bodyWriter.write(buff[0..(last_read > 
BUFF_SIZE ? 1024 : last_read)]);

                 if (last_read < BUFF_SIZE)
                 {
                     break;
                 }
             }
         }
     );
}

static this()
{
     setLogLevel(LogLevel.trace);

     auto settings = new HTTPServerSettings;
     settings.hostName = "localhost";
     settings.port = 8004;
     settings.bindAddresses = ["127.0.0.1"];

     auto router = new URLRouter;
     router.get("/download/*", toDelegate(&handle));

     listenHTTP(settings, router);
}
---

"Dicebot" <public dicebot.lv> writes:

 But it still has some issues. It works great for small files 
 but for larger
 (~150MB) VibeD just closes remote connection and web browsers 
 just keep trying to receive something until it timeouts.

 Another issue which will probably come is the fact that Vibe.D 
 won't be able to serve more than 1 client at the same time. At 
 least my attempts with 2MB file resulted with: 1st client 
 downloading file, 2nd client getting 500 from vibe.d

Such behavior may happen if you use some blocking I/O routine 
which results in timeout for pending connections as thread is 
blocked. However, built-in listenHTTP should be called 
asynchronously. Maybe you have found a bug, maybe something else 
is happening behind the hood, pretty hard to say without 
debugging.

Note that while you can add more worker threads since last vibe.d 
release, it is not really a solution as it just hides the problem 
instead of solving it - in proper event-based application single 
fiber should never block thread long enough to cause dropping of 
other requests.

I have not noticed anything fundamentally wrong with your snippet 
so some deep debugging is needed.

"Dicebot" <public dicebot.lv> writes:

Though using 1Kb buffer to read/write large file _may_ be an 
issue that results in such long processing time.

tl; dr: you need to make sure that no operation in your request 
handler takes more than fraction of second to complete.