www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Tracing allocations with "new" in dmd

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
dmd does most of its allcoation with new. I wonder what would be the 
easiest way to track how many instances of each type are created.

There are over 3000 call sites, so ideally the changes to the source 
would be entirely automatic (e.g. with sed).

Would be in your debt for any ideas. Thanks!
Jun 19 2020
next sibling parent reply Guillaume Piolat <first.name gmail.com> writes:
On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu 
wrote:
 dmd does most of its allcoation with new. I wonder what would 
 be the easiest way to track how many instances of each type are 
 created.

 There are over 3000 call sites, so ideally the changes to the 
 source would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
-profile=gc ?
Jun 19 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/19/20 9:48 AM, Guillaume Piolat wrote:
 On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:
 dmd does most of its allcoation with new. I wonder what would be the 
 easiest way to track how many instances of each type are created.

 There are over 3000 call sites, so ideally the changes to the source 
 would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
-profile=gc ?
Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?
Jun 19 2020
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 19 June 2020 at 14:19:36 UTC, Andrei Alexandrescu 
wrote:
 On 6/19/20 9:48 AM, Guillaume Piolat wrote:
 On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu 
 wrote:
 dmd does most of its allcoation with new. I wonder what would 
 be the easiest way to track how many instances of each type 
 are created.

 There are over 3000 call sites, so ideally the changes to the 
 source would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
-profile=gc ?
Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?
gc profiling will not work UNLESS you use the `-lowmem` switch.
Jun 19 2020
prev sibling parent reply Guillaume Piolat <first.name gmail.com> writes:
On Friday, 19 June 2020 at 14:19:36 UTC, Andrei Alexandrescu 
wrote:
 
 -profile=gc ?
Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?
Here is an example of a profilegc.log file: --------------- profilegc.log ----------- bytes allocated, allocations, type, function, file:line 4096 1 char[] D main source\main.d:36 240 10 main.main.D D main source\main.d:44 64 1 int[] D main source\main.d:37 56 1 double[] D main source\main.d:55 -----------------------------------------
Jun 19 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/19/20 10:34 AM, Guillaume Piolat wrote:
 On Friday, 19 June 2020 at 14:19:36 UTC, Andrei Alexandrescu wrote:
 -profile=gc ?
Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?
Here is an example of a profilegc.log file: --------------- profilegc.log ----------- bytes allocated, allocations, type, function, file:line            4096                  1    char[] D main source\main.d:36             240                 10    main.main.D D main source\main.d:44              64                  1    int[] D main source\main.d:37              56                  1    double[] D main source\main.d:55 -----------------------------------------
Thanks. A quick experiment (built dmd itself with -profile=gc, ran it on a few simple files) seems to show that only array allocations are traced, but not allocations of individual objects.
Jun 19 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/19/2020 8:36 AM, Andrei Alexandrescu wrote:
 A quick experiment (built dmd itself with -profile=gc, ran it on a few 
 simple files) seems to show that only array allocations are traced, but not 
 allocations of individual objects.
Please post to bugzilla.
Jun 19 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/19/20 4:59 PM, Walter Bright wrote:
 On 6/19/2020 8:36 AM, Andrei Alexandrescu wrote:
 A quick experiment (built dmd itself with -profile=gc, ran it on a few 
 simple files) seems to show that only array allocations are traced, 
 but not allocations of individual objects.
Please post to bugzilla.
https://issues.dlang.org/show_bug.cgi?id=20960
Jun 19 2020
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu 
wrote:
 dmd does most of its allcoation with new. I wonder what would 
 be the easiest way to track how many instances of each type are 
 created.

 There are over 3000 call sites, so ideally the changes to the 
 source would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
I have a tracing tool. USE IT.
Jun 19 2020
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 19 June 2020 at 14:23:10 UTC, Stefan Koch wrote:
 On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu 
 wrote:
 dmd does most of its allcoation with new. I wonder what would 
 be the easiest way to track how many instances of each type 
 are created.

 There are over 3000 call sites, so ideally the changes to the 
 source would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
I have a tracing tool. USE IT.
it's here: https://github.com/UplinkCoder/dmd/tree/dmd_tracing_2092 Currently I am still hunting down a bug which seems to be caused, By tracing a temporary allocated, expression which will then lead to a segfault while writing the trace. But I would very much like for others to use it, it's designed to allow you to say which source_code takes the most time out of the front-end process and where. You will have to modify the source code of dmd to investigate particular issues, But it comes with a number of likely candidates pre-selected for tracing. I am happy to work with someone on up-streaming this and making it better.
Jun 19 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/19/20 10:23 AM, Stefan Koch wrote:
 On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:
 dmd does most of its allcoation with new. I wonder what would be the 
 easiest way to track how many instances of each type are created.

 There are over 3000 call sites, so ideally the changes to the source 
 would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
I have a tracing tool. USE IT.
Cool, some links to docs would be great!
Jun 19 2020
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 19 June 2020 at 14:42:27 UTC, Andrei Alexandrescu 
wrote:
 On 6/19/20 10:23 AM, Stefan Koch wrote:
 On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu 
 wrote:
 dmd does most of its allcoation with new. I wonder what would 
 be the easiest way to track how many instances of each type 
 are created.

 There are over 3000 call sites, so ideally the changes to the 
 source would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
I have a tracing tool. USE IT.
Cool, some links to docs would be great!
There are no docs unfortunately. I am happy to provide assistance and in the process of exploration the docs will be created as well.
Jun 19 2020
prev sibling parent reply Tim <tim.dlang t-online.de> writes:
On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu 
wrote:
 dmd does most of its allcoation with new. I wonder what would 
 be the easiest way to track how many instances of each type are 
 created.

 There are over 3000 call sites, so ideally the changes to the 
 source would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers. I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications. dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d. For dmd you can call it like this: heaptrack dmd -lowmem test.d It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.
Jun 19 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/19/20 12:03 PM, Tim wrote:
 On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:
 dmd does most of its allcoation with new. I wonder what would be the 
 easiest way to track how many instances of each type are created.

 There are over 3000 call sites, so ideally the changes to the source 
 would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers. I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications. dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d. For dmd you can call it like this: heaptrack dmd -lowmem test.d It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.
Thanks. Does it provide type information about the objects allocated?
Jun 19 2020
next sibling parent Tim <tim.dlang t-online.de> writes:
On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu 
wrote:
 Thanks. Does it provide type information about the objects 
 allocated?
No, it does not save type information. It generates a stack trace for every allocation and can combine those into a bottom-up or top-down tree.
Jun 19 2020
prev sibling parent reply kinke <noone nowhere.com> writes:
On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu 
wrote:
 Does it provide type information about the objects allocated?
All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/ro t/rmem.d#L222-L310. There you have the TypeInfos.
Jun 19 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/19/20 6:05 PM, kinke wrote:
 On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:
 Does it provide type information about the objects allocated?
All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/ro t/rmem.d#L222-L310. There you have the TypeInfos.
Thank you! Just to clarify: this is advice on where to insert my own tracing, not a pointer to an existing instrumentation, right?
Jun 19 2020
parent kinke <noone nowhere.com> writes:
On Saturday, 20 June 2020 at 01:20:02 UTC, Andrei Alexandrescu 
wrote:
 On 6/19/20 6:05 PM, kinke wrote:
 On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu 
 wrote:
 Does it provide type information about the objects allocated?
All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/ro t/rmem.d#L222-L310. There you have the TypeInfos.
Thank you! Just to clarify: this is advice on where to insert my own tracing, not a pointer to an existing instrumentation, right?
Yes, instrumentation to be inserted.
Jun 19 2020
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 19 June 2020 at 16:03:55 UTC, Tim wrote:
 On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu 
 wrote:
 dmd does most of its allcoation with new. I wonder what would 
 be the easiest way to track how many instances of each type 
 are created.

 There are over 3000 call sites, so ideally the changes to the 
 source would be entirely automatic (e.g. with sed).

 Would be in your debt for any ideas. Thanks!
You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers. I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications. dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d. For dmd you can call it like this: heaptrack dmd -lowmem test.d It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.
Reply to bump. This is one of the best "secret weapons" in the D ecosystem. Just yesterday we used it to find the reason why a service was ballooning to 11GB.
Nov 10 2022
parent Tejas <notrealemail gmail.com> writes:
On Friday, 11 November 2022 at 07:26:37 UTC, FeepingCreature 
wrote:
 On Friday, 19 June 2020 at 16:03:55 UTC, Tim wrote:
 [...]
Reply to bump. This is one of the best "secret weapons" in the D ecosystem. Just yesterday we used it to find the reason why a service was ballooning to 11GB.
And of course it's not on `dub`, hurting adoption...
Nov 11 2022