www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - vibe.d maxRequestSize

reply Chris <wendlec tcd.ie> writes:
The vibe.d server rejects `XMLHttpRequest`s that are too long (in 
the eyes of the server). In the docs it says

"maxRequestSize   ulong

Maximum number of transferred bytes per request after which the 
connection is closed with [sic!]"

However, when you go to

http://vibed.org/api/vibe.http.server/HTTPServerSettings.maxRequestSize

it says

"Maximum number of transferred bytes per request after which the 
connection is closed with an error; not supported yet"

"not supported yet" and I think it is not supported yet. Can I 
still change the server settings so that it accepts longer query 
strings? Atm, the server rejects strings that are > ~2,500 
characters, which is not much.
Sep 14 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 9/14/16 9:58 AM, Chris wrote:
 The vibe.d server rejects `XMLHttpRequest`s that are too long (in the
 eyes of the server). In the docs it says

 "maxRequestSize   ulong

 Maximum number of transferred bytes per request after which the
 connection is closed with [sic!]"

 However, when you go to

 http://vibed.org/api/vibe.http.server/HTTPServerSettings.maxRequestSize

 it says

 "Maximum number of transferred bytes per request after which the
 connection is closed with an error; not supported yet"

 "not supported yet" and I think it is not supported yet. Can I still
 change the server settings so that it accepts longer query strings? Atm,
 the server rejects strings that are > ~2,500 characters, which is not much.
Hm.. I have adjusted this in my project, and it works (set to 50M). Needed it for uploading large images. Using version 0.7.29 -Steve
Sep 14 2016
parent reply Chris <wendlec tcd.ie> writes:
On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven 
Schveighoffer wrote:

 Hm.. I have adjusted this in my project, and it works (set to 
 50M). Needed it for uploading large images.

 Using version 0.7.29

 -Steve
It doesn't seem to make any difference in my case. I wonder what could be wrong
Sep 15 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 9/15/16 9:11 AM, Chris wrote:
 On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer
 wrote:

 Hm.. I have adjusted this in my project, and it works (set to 50M).
 Needed it for uploading large images.

 Using version 0.7.29

 -Steve
It doesn't seem to make any difference in my case. I wonder what could be wrong
Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -Steve
Sep 15 2016
next sibling parent reply Chris <wendlec tcd.ie> writes:
On Thursday, 15 September 2016 at 13:26:48 UTC, Steven 
Schveighoffer wrote:
 On 9/15/16 9:11 AM, Chris wrote:
 On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven 
 Schveighoffer
 wrote:

 Hm.. I have adjusted this in my project, and it works (set to 
 50M).
 Needed it for uploading large images.

 Using version 0.7.29

 -Steve
It doesn't seem to make any difference in my case. I wonder what could be wrong
Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -Steve
Hm, maybe the browsers are limiting the the length of the query string themselves, although that shouldn't be an issue with POST. I can send very long strings, if I enable HTTPServerOption.parseFormBody, but then I can no longer parse the URI at the same time (as far as I know). The browser always sends `Content-Length: 0` in the request header, which doesn't affect vibe.d (short strings work).
Sep 15 2016
parent reply sarn <sarn theartofmachinery.com> writes:
I hope this isn't too obvious, but I have to ask because it's 
such a common gotcha:

Are you reverse proxying through a server like nginx by any 
chance?  There are default request size limits there.  (For nginx 
specifically, it's this one:
https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size)
Sep 15 2016
parent Chris <wendlec tcd.ie> writes:
On Friday, 16 September 2016 at 00:35:25 UTC, sarn wrote:
 I hope this isn't too obvious, but I have to ask because it's 
 such a common gotcha:

 Are you reverse proxying through a server like nginx by any 
 chance?  There are default request size limits there.  (For 
 nginx specifically, it's this one:
 https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size)
No. Atm I'm running the server locally on my machine (for development), so I access vibe.d directly. But later it will run on Apache, and query string size will inevitably be an issue.
Sep 16 2016
prev sibling parent reply Chris <wendlec tcd.ie> writes:
On Thursday, 15 September 2016 at 13:26:48 UTC, Steven 
Schveighoffer wrote:
 On 9/15/16 9:11 AM, Chris wrote:
 On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven 
 Schveighoffer
 wrote:

 Hm.. I have adjusted this in my project, and it works (set to 
 50M).
 Needed it for uploading large images.

 Using version 0.7.29

 -Steve
It doesn't seem to make any difference in my case. I wonder what could be wrong
Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832 -Steve
Finally, I could find where it happens. If I use "get" in a HTML form (instead of "post"), vibe.d complains. I set the max request and request header size to 25MB. ` 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/strea /operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- ` https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/stream/operations.d#L349
Sep 19 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 9/19/16 11:59 AM, Chris wrote:
 On Thursday, 15 September 2016 at 13:26:48 UTC, Steven Schveighoffer wrote:
 On 9/15/16 9:11 AM, Chris wrote:
 On Wednesday, 14 September 2016 at 20:23:09 UTC, Steven Schveighoffer
 wrote:

 Hm.. I have adjusted this in my project, and it works (set to 50M).
 Needed it for uploading large images.

 Using version 0.7.29
It doesn't seem to make any difference in my case. I wonder what could be wrong
Seems to be in use: https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d#L1832
Finally, I could find where it happens. If I use "get" in a HTML form (instead of "post"), vibe.d complains. I set the max request and request header size to 25MB. ` 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- ` https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/stream/operations.d#L349
Hm... you don't get a full stack trace? Hard to tell what max_bytes should be, it defaults to ulong.max, so no way you are exhausting that. Without knowing where readUntilSmall is called, it's hard to diagnose. -Steve
Sep 19 2016
parent reply Chris <wendlec tcd.ie> writes:
On Monday, 19 September 2016 at 16:55:05 UTC, Steven 
Schveighoffer wrote:

 Hm... you don't get a full stack trace? Hard to tell what 
 max_bytes should be, it defaults to ulong.max, so no way you 
 are exhausting that. Without knowing where readUntilSmall is 
 called, it's hard to diagnose.

 -Steve
I didn't want to litter the thread, here it is: 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/strea /operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos std/exception.d:405 pure safe void std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, const(char[])) [0x8f11a7] /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos std/exception.d:363 pure safe bool std.exception.enforce!(Exception, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x8f112a] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:348 void vibe.stream.operations.readUntilSmall!(vibe.utils.array.All cAppender!(ubyte[], ubyte).AllocAppender).readUntilSmall(vibe.core stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06b00] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:137 void vibe.stream.operations.readUntil!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntil(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06a1f] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:53 void vibe.stream.operations.readLine!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readLine(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, ulong, immutable(char)[]) [0xb069c8] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:39 ubyte[] vibe.stream.operations.readLine!().readLine(vibe.core stream.InputStream, ulong, immutable(char)[], vibe.utils.memory.Allocator) [0xb06984] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1861 void vibe.http.server.parseRequestHeader(vibe.http.server.HTTPServerRequest, vibe.core.stream.InputStream, vibe.utils.memory.Allocator, ulong) [0xb5e827] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1659 bool vibe.http.server.handleRequest(vibe.core.stream.Stream, vibe.core.net.TCPConnection, vibe.http.server.HTTPListenInfo, ref vibe.http.server.HTTPServerSettings, ref bool) [0xb5c5d0] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1552 void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPListenInfo) [0xb5be4e] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1433 void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings).doListen(vibe.http.se ver.HTTPListenInfo, bool, bool).__lambda4(vibe.core.net.TCPConnection) [0xb5b814] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/drivers libevent2_tcp.d:610 void vibe.core.drivers.libevent2_tcp.ClientTask.execute() [0xbe8715] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/core.d:488 void vibe.core.core.makeTaskFuncInfo!(void delegate()).makeTaskFuncInfo(ref void delegate()).callDelegate(vibe.core.core.TaskFuncInfo*) [0xafea75] ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/core/core.d:1119 void vibe.core.core.CoreTask.run() [0xb8dd79] ??:? void core.thread.Fiber.run() [0xc70ea5] ??:? fiber_entryPoint [0xc70c27] ??:? [0xffffffff]
Sep 19 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 9/19/16 1:34 PM, Chris wrote:
 On Monday, 19 September 2016 at 16:55:05 UTC, Steven Schveighoffer wrote:

 Hm... you don't get a full stack trace? Hard to tell what max_bytes
 should be, it defaults to ulong.max, so no way you are exhausting
 that. Without knowing where readUntilSmall is called, it's hard to
 diagnose.

 -Steve
I didn't want to litter the thread, here it is: 400 - Bad Request Bad Request Internal error information: object.Exception ../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d(349): Reached maximum number of bytes while searching for end marker. ---------------- /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:405 pure safe void std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, const(char[])) [0x8f11a7] /home/chris/.dvm/compilers/dmd-2.071.1/linux/bin/../../src/phobos/std/exception.d:363 pure safe bool std.exception.enforce!(Exception, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong) [0x8f112a] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:348 void vibe.stream.operations.readUntilSmall!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntilSmall(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06b00] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:137 void vibe.stream.operations.readUntil!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readUntil(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, const(ubyte[]), ulong) [0xb06a1f] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:53 void vibe.stream.operations.readLine!(vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender).readLine(vibe.core.stream.InputStream, ref vibe.utils.array.AllocAppender!(ubyte[], ubyte).AllocAppender, ulong, immutable(char)[]) [0xb069c8] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/stream/operations.d:39 ubyte[] vibe.stream.operations.readLine!().readLine(vibe.core.stream.InputStream, ulong, immutable(char)[], vibe.utils.memory.Allocator) [0xb06984] .../../.dub/packages/vibe-d-0.7.29/vibe-d/source/vibe/http/server.d:1861 void vibe.http.server.parseRequestHeader(vibe.http.server.HTTPServerRequest, vibe.core.stream.InputStream, vibe.utils.memory.Allocator, ulong) [0xb5e827]
Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -Steve
Sep 19 2016
parent reply Chris <wendlec tcd.ie> writes:
On Monday, 19 September 2016 at 17:54:05 UTC, Steven 
Schveighoffer wrote:
 On 9/19/16 1:34 PM, Chris wrote:
 [...]
Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -Steve
Thanks a million! I'll look into it tomorrow.
Sep 19 2016
parent Chris <wendlec tcd.ie> writes:
On Monday, 19 September 2016 at 18:13:12 UTC, Chris wrote:
 On Monday, 19 September 2016 at 17:54:05 UTC, Steven 
 Schveighoffer wrote:
 On 9/19/16 1:34 PM, Chris wrote:
 [...]
Here is the culprit: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 And the definition of MaxHTTPHeaderLineLength is: https://github.com/rejectedsoftware/vibe.d/blob/0.7.29/source/vibe/http/server.d#L1861 4096 bytes. I'm not super-familiar with HTTP protocol requirements, but a further diagnosis of your net traffic may reveal that your browser is doing something unorthodox, or vibe needs to adjust here. -Steve
Thanks a million! I'll look into it tomorrow.
FYI, I had to solve the issue by sending a form via a JS XMLHTTPRequest. maxRequestSize didn't help.
Sep 22 2016