www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - High memory usage in vibe.d application

reply Anton Fediushin <fediushin.anton yandex.com> writes:
Hello, I'm looking for an advice on what I am doing wrong.

I have a vibe.d-based program, which connects to an audio stream 
and gets name of the song currently playing. For that, I wrote 
the following code:

```
 safe string nowPlaying(string url) {
	import vibe.core.stream;
	string r;
	url.requestHTTP(
		(scope req) {
			req.headers.addField("Icy-MetaData", "1");
		},
		(scope res) {
			auto metaint = res.headers.get("icy-metaint").to!int;
			auto buffer = new ubyte[metaint];
			res.bodyReader.read(buffer, IOMode.all);
			
			auto lengthBuff = new ubyte[1];
			res.bodyReader.read(lengthBuff, IOMode.all);

			auto dataBuffer = new ubyte[lengthBuff[0] * 16];
			res.bodyReader.read(dataBuffer, IOMode.all);
			r = dataBuffer.map!(a => 
a.to!char).split('\'').drop(1).front.array.idup;
		}
	);
	return r;
}
```

And I call it with a timer every 10 seconds:

```
string now_playing;
10.seconds.setTimer(() {
	now_playing = nowPlaying(stream);
}, true);
```

This code worked fine for 8 or so hours and then got killed by 
docker because of a limit of 64MB of RAM. I executed the same 
code on my machine and saw resident set size growing in 
real-time. Blaming GC (as people usually do) I changed the code 
to use std.experimental.allocator instead:

```
 safe string nowPlaying(string url) {
	import vibe.core.stream;
	import std.experimental.allocator;
	string r;
	url.requestHTTP(
		(scope req) {
			req.headers.addField("Icy-MetaData", "1");
		},
		(scope res) {
			auto metaint = res.headers.get("icy-metaint").to!int;
			auto buffer = theAllocator.makeArray!ubyte(metaint);
			scope(exit) theAllocator.dispose(buffer);
			res.bodyReader.read(buffer, IOMode.all);
			
			auto lengthBuffer = theAllocator.makeArray!ubyte(1);
			scope(exit) theAllocator.dispose(lengthBuffer);
			res.bodyReader.read(lengthBuffer, IOMode.all);

			auto dataBuffer = theAllocator.makeArray!ubyte(lengthBuffer[0] 
* 16);
			scope(exit) theAllocator.dispose(dataBuffer);
			res.bodyReader.read(dataBuffer, IOMode.all);

			r = dataBuffer.map!(a => 
a.to!char).split('\'').drop(1).front.array.idup;
		}
	);
	return r;
}
```

And somehow, it got *worse*. Now my program gets killed every 3 
hours. How is that possible? Am I missing something?

Some screenshots of CPU/Memory usage:
1. These are metrics of a whole cluster, program is started at 
around 8:00 and gets killed after 16:00 
https://imgur.com/a/IhHvOt4
2. These are metrics of an updated program which uses 
std.experimental.allocator. https://imgur.com/a/XBchJ7C
Jun 29 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
Almost forgot, there are two timers which call this function for 
two different streams.

Value of `metaint` is 16000, which means that only 16KB of memory 
are allocated for the `buffer`, then it reads another byte which 
contains length of the metadata / 16 and then it reads the 
metadata which is 100-200 bytes long.

This gives us... 16KiB per one nowPlaying() call. Why doesn't it 
free the memory?
Jun 29 2018
parent reply Radu <void null.pt> writes:
On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote:

 Almost forgot, there are two timers which call this function 
 for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of 
 memory are allocated for the `buffer`, then it reads another 
 byte which contains length of the metadata / 16 and then it 
 reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why doesn't 
 it free the memory?


Maybe use the 
https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of
theAllocator as it defaults to GC.

Also, why you .idup the array? .array already creates a new one 
on the heap.
Jun 29 2018
next sibling parent reply bauss <jj_1337 live.dk> writes:
On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:

 On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote:

 Almost forgot, there are two timers which call this function 
 for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of 
 memory are allocated for the `buffer`, then it reads another 
 byte which contains length of the metadata / 16 and then it 
 reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why doesn't 
 it free the memory?


 Maybe use the 
 https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of
theAllocator as it defaults to GC.

 Also, why you .idup the array? .array already creates a new one 
 on the heap.


This.

Which kind of makes the usage of theAllocator useless.

I was going to suggest using  nogc too, because it would most 
likely be GC allocated memory that is taking up space.

I run multiple vibe.d applications and I have no issues with 
memory (Even with GC.)
Jun 29 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 10:31:14 UTC, bauss wrote:

 On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:

 On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote:

 Almost forgot, there are two timers which call this function 
 for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of 
 memory are allocated for the `buffer`, then it reads another 
 byte which contains length of the metadata / 16 and then it 
 reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why doesn't 
 it free the memory?


 Maybe use the 
 https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of
theAllocator as it defaults to GC.

 Also, why you .idup the array? .array already creates a new 
 one on the heap.


 This.

 Which kind of makes the usage of theAllocator useless.


Indeed, because it uses GC by default my `theAllocator.dispose` 
did nothing, which basically made these two samples of code equal.


 I was going to suggest using  nogc too, because it would most 
 likely be GC allocated memory that is taking up space.


It is GC's fault for sure, I built my program with profile-gc and 
it allocated a lot there. Question is, why doesn't it free this 
memory?


 I run multiple vibe.d applications and I have no issues with 
 memory (Even with GC.)


Me neither, my other vibe.d project uses 7.5MB and that's it.
Jun 29 2018
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 29/06/2018 11:09 PM, Anton Fediushin wrote:

 It is GC's fault for sure, I built my program with profile-gc and it 
 allocated a lot there. Question is, why doesn't it free this memory?


Probably doesn't know that it should deallocate so eagerly.
A GC.collect(); call may help.
Jun 29 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote:

 On 29/06/2018 11:09 PM, Anton Fediushin wrote:

 It is GC's fault for sure, I built my program with profile-gc 
 and it allocated a lot there. Question is, why doesn't it free 
 this memory?


 Probably doesn't know that it should deallocate so eagerly.
 A GC.collect(); call may help.


That's a good idea. GC really needs to be kicked in once in a 
while because it did _nothing_ in 8 hours, even though my 
application is just a couple of timers - it isn't a hard task for 
CPU or memory and there's plenty of time to collect some garbage.

Now I finally understand why GC is not a great thing. I was 
writing apps utilizing GC for a long time and never had problems 
with it, but when it came down to this simple program it stabbed 
me in the back.
Jun 29 2018
next sibling parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote:

 On 29/06/2018 11:09 PM, Anton Fediushin wrote:

 It is GC's fault for sure, I built my program with profile-gc 
 and it allocated a lot there. Question is, why doesn't it 
 free this memory?


 Probably doesn't know that it should deallocate so eagerly.
 A GC.collect(); call may help.


 That's a good idea. GC really needs to be kicked in once in a 
 while because it did _nothing_ in 8 hours, even though my 
 application is just a couple of timers - it isn't a hard task 
 for CPU or memory and there's plenty of time to collect some 
 garbage.

 Now I finally understand why GC is not a great thing. I was 
 writing apps utilizing GC for a long time and never had 
 problems with it, but when it came down to this simple program 
 it stabbed me in the back.


Which language that you had write apps in that utilize GC? Java? 


Alexander
Jun 29 2018
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 16:19:39 UTC, 12345swordy wrote:

 On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:

 Now I finally understand why GC is not a great thing. I was 
 writing apps utilizing GC for a long time and never had 
 problems with it, but when it came down to this simple program 
 it stabbed me in the back.


 Which language that you had write apps in that utilize GC? 

 GC.

 Alexander


Talking about D here.

GC can be the best option for some languages and environments, 
but it doesn't fit D that well. Writing programs in D I always 
know where stack-allocated structs get deleted and such, but I 
have no idea on what's going on with the GC. Does it collect 
anything at all? Why doesn't it collect this? How do I force it 
to collect this?
Jun 29 2018
prev sibling parent reply Bauss <jj_1337 live.dk> writes:
On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole wrote:

 On 29/06/2018 11:09 PM, Anton Fediushin wrote:

 It is GC's fault for sure, I built my program with profile-gc 
 and it allocated a lot there. Question is, why doesn't it 
 free this memory?


 Probably doesn't know that it should deallocate so eagerly.
 A GC.collect(); call may help.


 That's a good idea. GC really needs to be kicked in once in a 
 while because it did _nothing_ in 8 hours, even though my 
 application is just a couple of timers - it isn't a hard task 
 for CPU or memory and there's plenty of time to collect some 
 garbage.

 Now I finally understand why GC is not a great thing. I was 
 writing apps utilizing GC for a long time and never had 
 problems with it, but when it came down to this simple program 
 it stabbed me in the back.


I wouldn't really blame the GC. There is a higher chance you're 
just not using it how it's meant to be, especially since it looks 
like you're mixing manual memory management with GC memory.
Jun 29 2018
next sibling parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 16:49:41 UTC, Bauss wrote:

 On Friday, 29 June 2018 at 16:07:00 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 11:11:57 UTC, rikki cattermole 
 wrote:

 On 29/06/2018 11:09 PM, Anton Fediushin wrote:

 It is GC's fault for sure, I built my program with 
 profile-gc and it allocated a lot there. Question is, why 
 doesn't it free this memory?


 Probably doesn't know that it should deallocate so eagerly.
 A GC.collect(); call may help.


 That's a good idea. GC really needs to be kicked in once in a 
 while because it did _nothing_ in 8 hours, even though my 
 application is just a couple of timers - it isn't a hard task 
 for CPU or memory and there's plenty of time to collect some 
 garbage.

 Now I finally understand why GC is not a great thing. I was 
 writing apps utilizing GC for a long time and never had 
 problems with it, but when it came down to this simple program 
 it stabbed me in the back.


 I wouldn't really blame the GC. There is a higher chance you're 
 just not using it how it's meant to be, especially since it 
 looks like you're mixing manual memory management with GC 
 memory.


I am not quite sure what should I blame now, because even if I 
use malloc for memory allocation, memory goes... somewhere?

So, long story short:
- Usage of Mallocator instead of theAllocator made it a little 
bit better
- VibeManualMemoryManagement had no (or little) effect
- Manually calling GC.collect had no (or little) effect

It makes me think that error is somewhere else. I made a code 
snippet of my testing program: https://gitlab.com/snippets/1729304

There are some changes to it:
- It uses different stream with metaint of 32KB
- It calls nowPlaying() every second

Now I will take a break from this, dealing with this kind of 
nonsense annoys me.
Jun 29 2018
parent reply Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Friday, 29 June 2018 at 17:40:07 UTC, Anton Fediushin wrote:

 So, long story short:
 - Usage of Mallocator instead of theAllocator made it a little 
 bit better
 - VibeManualMemoryManagement had no (or little) effect
 - Manually calling GC.collect had no (or little) effect


You could try to call GC.minimize in pair with GC.collect:

```
GC.collect();
GC.minimize();
```

to return all freed memory back to the OS.

Not sure that the leakage of this type is possible because if 
you're running your program on 64bit Linux the probability of it 
is very low. AFAIK the GC is launched every (almost) time you 
allocate the memory, and if it finds "dead" pointers, it 
definitely must clean them out.

Vibe.d may also leak. Have you tried to run the same code without 
Vibe.d, say, using https://github.com/ikod/dlang-requests as an 
HTTP client?

Also, have you tried to change vibe.d's event loop engine, like 
libevent or libasync?
Jun 30 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Saturday, 30 June 2018 at 22:06:50 UTC, Jacob Shtokolov wrote:

 On Friday, 29 June 2018 at 17:40:07 UTC, Anton Fediushin wrote:

 So, long story short:
 - Usage of Mallocator instead of theAllocator made it a little 
 bit better
 - VibeManualMemoryManagement had no (or little) effect
 - Manually calling GC.collect had no (or little) effect


 You could try to call GC.minimize in pair with GC.collect:

 ```
 GC.collect();
 GC.minimize();
 ```

 to return all freed memory back to the OS.



With vibe.d this had no effect too.


 Not sure that the leakage of this type is possible because if 
 you're running your program on 64bit Linux the probability of 
 it is very low. AFAIK the GC is launched every (almost) time 
 you allocate the memory, and if it finds "dead" pointers, it 
 definitely must clean them out.

 Vibe.d may also leak. Have you tried to run the same code 
 without Vibe.d, say, using 
 https://github.com/ikod/dlang-requests as an HTTP client?


Now I tried it and indeed, it's vibe.d's fault. I'm not quite 
sure what causes it and if this problem is known, I'll look into 
that later and open an issue if it doesn't exist already.
Jun 30 2018
parent reply Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Sunday, 1 July 2018 at 05:20:17 UTC, Anton Fediushin wrote:

 Now I tried it and indeed, it's vibe.d's fault. I'm not quite 
 sure what causes it and if this problem is known, I'll look 
 into that later and open an issue if it doesn't exist already.


Yes, please do this when you have time. That would be really 
helpful for further vibe.d improvement.

I remember a pretty old (and closed) bug of HTTP client here:
https://github.com/vibe-d/vibe.d/issues/1321

So it might be somehow related to this one. Probably something 
wrong with HTTP client or TLS/SSL related logic. You example code 
is very good and I was able to reproduce the same issue with the 
latest stable compiler, so I hope the guys will find the problem.

Thanks!
Jul 01 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Sunday, 1 July 2018 at 12:32:25 UTC, Jacob Shtokolov wrote:

 On Sunday, 1 July 2018 at 05:20:17 UTC, Anton Fediushin wrote:

 Now I tried it and indeed, it's vibe.d's fault. I'm not quite 
 sure what causes it and if this problem is known, I'll look 
 into that later and open an issue if it doesn't exist already.


 Yes, please do this when you have time. That would be really 
 helpful for further vibe.d improvement.

 I remember a pretty old (and closed) bug of HTTP client here:
 https://github.com/vibe-d/vibe.d/issues/1321

 So it might be somehow related to this one. Probably something 
 wrong with HTTP client or TLS/SSL related logic. You example 
 code is very good and I was able to reproduce the same issue 
 with the latest stable compiler, so I hope the guys will find 
 the problem.

 Thanks!


I reduced the test case to _one_ line:
```
1.seconds.setTimer(() => "http://google.com".requestHTTP((scope 
req) {}, (scope res) {res.disconnect;}), true);
```

What happens is `res.disconnect` doesn't free all of the internal 
buffers, causing leakage. One way to avoid that is to call 
`res.dropBody`, but it isn't always wanted (just like in my 
example).

I submitted an issue: https://github.com/vibe-d/vibe.d/issues/2179
Jul 01 2018
parent reply crimaniak <crimaniak gmail.com> writes:
On Sunday, 1 July 2018 at 13:44:23 UTC, Anton Fediushin wrote:


 I reduced the test case to _one_ line:
 ```
 1.seconds.setTimer(() => "http://google.com".requestHTTP((scope 
 req) {}, (scope res) {res.disconnect;}), true);
 ```

 What happens is `res.disconnect` doesn't free all of the 
 internal buffers, causing leakage. One way to avoid that is to 
 call `res.dropBody`, but it isn't always wanted (just like in 
 my example).


  The problem is known and mentioned in the documentation:
http://vibed.org/api/vibe.http.client/requestHTTP


Note that it is highly recommended to use one of the overloads 
that take a responder callback, as they can avoid some memory 
allocations and are safe against accidentally leaving stale 
response objects (objects whose response body wasn't fully 
read). For the returning overloads of the function it is 
recommended to put a scope(exit) right after the call in which 
HTTPClientResponse.dropBody is called to avoid this.


As I understand the situation, request object will reside in 
memory until you fully read or do something with response body.
Jul 01 2018
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Sunday, 1 July 2018 at 20:15:02 UTC, crimaniak wrote:

 On Sunday, 1 July 2018 at 13:44:23 UTC, Anton Fediushin wrote:


 I reduced the test case to _one_ line:
 ```
 1.seconds.setTimer(() => 
 "http://google.com".requestHTTP((scope req) {}, (scope res) 
 {res.disconnect;}), true);
 ```

 What happens is `res.disconnect` doesn't free all of the 
 internal buffers, causing leakage. One way to avoid that is to 
 call `res.dropBody`, but it isn't always wanted (just like in 
 my example).


  The problem is known and mentioned in the documentation:
 http://vibed.org/api/vibe.http.client/requestHTTP


Note that it is highly recommended to use one of the overloads 
that take a responder callback, as they can avoid some memory 
allocations and are safe against accidentally leaving stale 
response objects (objects whose response body wasn't fully 
read). For the returning overloads of the function it is 
recommended to put a scope(exit) right after the call in which 
HTTPClientResponse.dropBody is called to avoid this.


 As I understand the situation, request object will reside in 
 memory until you fully read or do something with response body.


It says so "for the returning overloads". Callback-based ones 
should be "safe against accidentally leaving stale response 
objects". Actually, in this example I don't 'accidentally' leave 
objects, I do that on purpose and call `res.disconnect` to 
forcefully close everything. Yet it still doesn't free memory.

There's nothing much to do with the response body - it can be 
either read and destroyed or just destroyed, and `res.disconect` 
should do this.
Jul 01 2018
prev sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 30/06/2018 4:49 AM, Bauss wrote:

 I wouldn't really blame the GC. There is a higher chance you're just not 
 using it how it's meant to be, especially since it looks like you're 
 mixing manual memory management with GC memory.


Let's be honest, I don't think it was meant to live in a container with 
64mb of ram. I just don't think it is kicking in to collect.
Jun 29 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Saturday, 30 June 2018 at 05:00:35 UTC, rikki cattermole wrote:

 On 30/06/2018 4:49 AM, Bauss wrote:

 I wouldn't really blame the GC. There is a higher chance 
 you're just not using it how it's meant to be, especially 
 since it looks like you're mixing manual memory management 
 with GC memory.


 Let's be honest, I don't think it was meant to live in a 
 container with 64mb of ram. I just don't think it is kicking in 
 to collect.


It doesn't, I'm experimenting with different GC configurations 
[1]. By default [2] `maxPoolSize` is set to 64MB, so maybe 
program gets killed by docker right before GC decides to collect.

[1] https://dlang.org/spec/garbage.html#gc_config
[2] 
https://github.com/dlang/druntime/blob/master/src/gc/config.d#L23
Jun 30 2018
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 30/06/2018 7:42 PM, Anton Fediushin wrote:

 On Saturday, 30 June 2018 at 05:00:35 UTC, rikki cattermole wrote:

 On 30/06/2018 4:49 AM, Bauss wrote:

 I wouldn't really blame the GC. There is a higher chance you're just 
 not using it how it's meant to be, especially since it looks like 
 you're mixing manual memory management with GC memory.


 Let's be honest, I don't think it was meant to live in a container 
 with 64mb of ram. I just don't think it is kicking in to collect.


 
 It doesn't, I'm experimenting with different GC configurations [1]. By 
 default [2] `maxPoolSize` is set to 64MB, so maybe program gets killed 
 by docker right before GC decides to collect.
 
 [1] https://dlang.org/spec/garbage.html#gc_config
 [2] https://github.com/dlang/druntime/blob/master/src/gc/config.d#L23


The OS ext. takes memory too.

34mb might be safer.
Jun 30 2018
prev sibling parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:

 On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote:

 Almost forgot, there are two timers which call this function 
 for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of 
 memory are allocated for the `buffer`, then it reads another 
 byte which contains length of the metadata / 16 and then it 
 reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why doesn't 
 it free the memory?


 Maybe use the 
 https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of
theAllocator as it defaults to GC.


Thanks, I'll try that.


 Also, why you .idup the array? .array already creates a new one 
 on the heap.


It does, but it creates char[] and I need a string. I changed 
code a little bit to remove unnecessary `map` and `idup` too.

Code now:

```
 safe string nowPlaying(string url) {
	import vibe.core.stream;
	import std.experimental.allocator;
	import std.experimental.allocator.mallocator;
	import std.string;

	string r;
	url.requestHTTP(
		(scope req) {
			req.headers.addField("Icy-MetaData", "1");
		},
		(scope res) {
			RCIAllocator a = allocatorObject(Mallocator.instance);

			auto metaint = res.headers.get("icy-metaint").to!int;
			auto buffer = a.makeArray!ubyte(metaint);
			scope(exit) a.dispose(buffer);
			res.bodyReader.read(buffer, IOMode.all);

			auto lengthBuffer = a.makeArray!ubyte(1);
			scope(exit) a.dispose(lengthBuffer);
			res.bodyReader.read(lengthBuffer, IOMode.all);

			auto dataBuffer = a.makeArray!ubyte(lengthBuffer[0] * 16);
			scope(exit) a.dispose(dataBuffer);
			res.bodyReader.read(dataBuffer, IOMode.all);

			r = dataBuffer.split('\'').drop(1).front.array.assumeUTF;

			res.disconnect;
		}
	);
	return r;
}
```

I will deploy that and see if it changes anything.
Jun 29 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:

 On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin wrote:

 Almost forgot, there are two timers which call this function 
 for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of 
 memory are allocated for the `buffer`, then it reads another 
 byte which contains length of the metadata / 16 and then it 
 reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why doesn't 
 it free the memory?


 Maybe use the 
 https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of
theAllocator as it defaults to GC.


 Thanks, I'll try that.
 ...
 I will deploy that and see if it changes anything.


It did! Memory usage went down to 7MiB yet it still grows 
slightly. I'll monitor if it changes in a couple of hours but it 
is much better.

Thank you a lot, Radu. It turns out that theAllocator is so 
tricky.
Jun 29 2018
parent reply bauss <jj_1337 live.dk> writes:
On Friday, 29 June 2018 at 11:24:14 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:

 On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin 
 wrote:

 Almost forgot, there are two timers which call this function 
 for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of 
 memory are allocated for the `buffer`, then it reads another 
 byte which contains length of the metadata / 16 and then it 
 reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why 
 doesn't it free the memory?


 Maybe use the 
 https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of
theAllocator as it defaults to GC.


 Thanks, I'll try that.
 ...
 I will deploy that and see if it changes anything.


 It did! Memory usage went down to 7MiB yet it still grows 
 slightly. I'll monitor if it changes in a couple of hours but 
 it is much better.

 Thank you a lot, Radu. It turns out that theAllocator is so 
 tricky.


Again you could do  nogc and see what memory is possibly 
allocated by the GC and perhaps that way you can see what memory 
the GC is holding on to.

non-GC memory should be freed right away and those there 
shouldn't be a leak from that.
Jun 29 2018
parent reply Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 11:42:18 UTC, bauss wrote:

 On Friday, 29 June 2018 at 11:24:14 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:

 On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin 
 wrote:

 Almost forgot, there are two timers which call this 
 function for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of 
 memory are allocated for the `buffer`, then it reads 
 another byte which contains length of the metadata / 16 and 
 then it reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why 
 doesn't it free the memory?


 Maybe use the 
 https://dlang.org/phobos/std_experimental_allocator_mallocator.html instead of
theAllocator as it defaults to GC.


 Thanks, I'll try that.
 ...
 I will deploy that and see if it changes anything.


 It did! Memory usage went down to 7MiB yet it still grows 
 slightly. I'll monitor if it changes in a couple of hours but 
 it is much better.

 Thank you a lot, Radu. It turns out that theAllocator is so 
 tricky.


 Again you could do  nogc and see what memory is possibly 
 allocated by the GC and perhaps that way you can see what 
 memory the GC is holding on to.


 nogc tells nothing new, just an error on every single line 
because neither `res.bodyReader.read` nor Mallocator's functions 
are marked as  nogc.

Compiling with dmd's `-vgc` flag shows nothing but the last line.


 non-GC memory should be freed right away and those there 
 shouldn't be a leak from that.


Using Mallocator instead of theAllocator improved the situation, 
but it still leaks for some reason. After 2 hours it went from 
7MiB to 18MiB.

I will compile it with profile-gc again and look for the possible 
cause of that, maybe I'll try valgrind too.
Jun 29 2018
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
Have you try use VibeManualMemoryManagement

https://github.com/TechEmpower/FrameworkBenchmarks/blob/3b24d0a21463edc536b30e2cea647fd425915401/frameworks/D/vibed/dub.json#L22


On Fri, Jun 29, 2018 at 3:20 PM Anton Fediushin via Digitalmars-d-learn <
digitalmars-d-learn puremagic.com> wrote:


 On Friday, 29 June 2018 at 11:42:18 UTC, bauss wrote:

 On Friday, 29 June 2018 at 11:24:14 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 11:01:41 UTC, Anton Fediushin wrote:

 On Friday, 29 June 2018 at 10:21:24 UTC, Radu wrote:

 On Friday, 29 June 2018 at 09:44:27 UTC, Anton Fediushin
 wrote:

 Almost forgot, there are two timers which call this
 function for two different streams.

 Value of `metaint` is 16000, which means that only 16KB of
 memory are allocated for the `buffer`, then it reads
 another byte which contains length of the metadata / 16 and
 then it reads the metadata which is 100-200 bytes long.

 This gives us... 16KiB per one nowPlaying() call. Why
 doesn't it free the memory?


 Maybe use the
 https://dlang.org/phobos/std_experimental_allocator_mallocator.html








 instead of theAllocator as it defaults to GC.

 Thanks, I'll try that.
 ...
 I will deploy that and see if it changes anything.


 It did! Memory usage went down to 7MiB yet it still grows
 slightly. I'll monitor if it changes in a couple of hours but
 it is much better.

 Thank you a lot, Radu. It turns out that theAllocator is so
 tricky.


 Again you could do  nogc and see what memory is possibly
 allocated by the GC and perhaps that way you can see what
 memory the GC is holding on to.


  nogc tells nothing new, just an error on every single line
 because neither `res.bodyReader.read` nor Mallocator's functions
 are marked as  nogc.

 Compiling with dmd's `-vgc` flag shows nothing but the last line.


 non-GC memory should be freed right away and those there
 shouldn't be a leak from that.


 Using Mallocator instead of theAllocator improved the situation,
 but it still leaks for some reason. After 2 hours it went from
 7MiB to 18MiB.

 I will compile it with profile-gc again and look for the possible
 cause of that, maybe I'll try valgrind too.
Jun 29 2018
parent Anton Fediushin <fediushin.anton yandex.com> writes:
On Friday, 29 June 2018 at 14:10:26 UTC, Daniel Kozak wrote:

 Have you try use VibeManualMemoryManagement

 https://github.com/TechEmpower/FrameworkBenchmarks/blob/3b24d0a21463edc536b30e2cea647fd425915401/frameworks/D/vibed/dub.json#L22



I'll try, not quite sure it'll help much.
Jun 29 2018