www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.experimental.allocator API and nearly useless StatsCollector

reply Luis <Luis.panadero gmail.com> writes:
Well, this last week I was playing using malloc/free to do some 
 nogc code, and using the std.experimental.allocator API. Also, I 
was trying to see a way to profile my code and search for memory 
leaks, etc...

What I initially did, was implementing my own allocator API plus 
my own mallocator, seeing how ECS buble engine does this.
Later, I take a look how std.experimental.allocator API works and 
I ended writing my own wrapper above make, makeArray, dispose, 
expandArray & shrinkArray that implements this profiler/stats, 
and using std.experimental.mallocator .
Finally, I catch that I can only use my wrapper on my code, and 
that I couldn't use on other third party libraries that allow to 
use allocators (like EMSI Containers), because (obviously) uses 
the std.experimental.allocator API.
Then, I discover 
std.experimental.allocator.building_blocks.stats_collector that 
looks that does what my littler profiler does, and more. But I 
see tiny detail that I don't see totally ok.

The stats that are reported by reportPerCallStatistics, shows the 
file & line number of the allocator!

Like this :

```
/usr/include/dmd/phobos/std/experimental/allocator/package.d(1585):
[numAllocate:1, numAllocateOK:1, bytesAllocated:40]
```

This could be useful for someone writing his own allocator, but 
it's pretty useless for anyone trying to track where a memory 
leak originated in his own code!

On my wrapper, What I did was to have a `string file = __FILE__, 
int line = __LINE__` parameters append to the make, makeArray, 
dispose, etc... functions and uses the file and & line to record 
where something was allocated/deallocated/reallocated.

So, why std.experimental.allocator API not does something similar 
to this ? The allocator implementations, would need to pass & 
ignore these two additional parameters. Except StatsCollector 
that can use it to show where the 
allocation/deallocation/reallocation it's happening
Jun 20
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:
 Well, this last week I was playing using malloc/free to do some 
  nogc code, and using the std.experimental.allocator API. Also, 
 I was trying to see a way to profile my code and search for 
 memory leaks, etc...

 [...]

 The stats that are reported by reportPerCallStatistics, shows 
 the file & line number of the allocator!

 Like this :

 ```
 /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585):
[numAllocate:1, numAllocateOK:1, bytesAllocated:40]
 ```

 This could be useful for someone writing his own allocator, but 
 it's pretty useless for anyone trying to track where a memory 
 leak originated in his own code!
To understand leaks I'd use valgrind instead. As it is based on DWARF info, reported leaking blocks are accompanied with a pretty (after demangling) back trace. So the first entry for each block will be unsurprinsingly always the same (e.g glibc malloc) but the other can be more interesting.
Jun 20
parent reply Luis <Luis.panadero gmail.com> writes:
On Sunday, 20 June 2021 at 16:53:06 UTC, Basile B. wrote:
 On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:
 Well, this last week I was playing using malloc/free to do 
 some  nogc code, and using the std.experimental.allocator API. 
 Also, I was trying to see a way to profile my code and search 
 for memory leaks, etc...

 [...]

 The stats that are reported by reportPerCallStatistics, shows 
 the file & line number of the allocator!

 Like this :

 ```
 /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585):
[numAllocate:1, numAllocateOK:1, bytesAllocated:40]
 ```

 This could be useful for someone writing his own allocator, 
 but it's pretty useless for anyone trying to track where a 
 memory leak originated in his own code!
To understand leaks I'd use valgrind instead. As it is based on DWARF info, reported leaking blocks are accompanied with a pretty (after demangling) back trace. So the first entry for each block will be unsurprinsingly always the same (e.g glibc malloc) but the other can be more interesting.
Yeah. I know Valgrind. I used it a lot, when I did C or C++ stuff, and it's a wonderful tool. However have his problems : 1. Makes anything to run painfull slow. 2. At least with DMD, the mangled information & stack traces are hard to follow. Compare this two outputs when I comment the code that calls to allocator.dispose(): My simple & dirty profiler : ``` $ ./bin/ddiv-test-unittest -t 1 -i "Stack" ✓ ddiv.container.stack SimpleStack Summary: 1 passed, 0 failed in 0 ms WARNING! Possible memory leaks! ----- Memory allocation information ----- Total amount of allocated memory that has not been released: 65536 bytes Addr: 0x55C70F4D0E60 - [int[]] 65536 bytes - source/ddiv/container/stack.d:84 ``` Valgrind (fun thing. I need to use --leak-check=full to see the memory leak of my code, at at 0x483DFAF realloc. The rest looks that it's from Silly and/or D runtime): ``` $ valgrind --leak-check=full ./bin/ddiv-test-unittest -t 1 -i "Stack" ==185906== Memcheck, a memory error detector ==185906== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==185906== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info ==185906== Command: ./bin/ddiv-test-unittest -t 1 -i Stack ==185906== --185906-- WARNING: Serious error when reading debug info --185906-- When reading debug info from /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest: --185906-- DWARF line info appears to be corrupt - the section is too small --185906-- WARNING: Serious error when reading debug info --185906-- When reading debug info from /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest: --185906-- read_filename_table: .debug_line is missing? ✓ ddiv.container.stack SimpleStack Summary: 1 passed, 0 failed in 89 ms ==185906== Conditional jump or move depends on uninitialised value(s) ==185906== at 0x44B2C8: _D4core8internal2gc4impl12conservativeQw3Gcx__T4markVbi0Vbi0ZQoMFNbNlSQCqQCoQCiQCiQCgQCrQBw__ 9ScanRangeVbi0ZQpZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x445469: _D4core8internal2gc4impl12conservativeQw3Gcx16markConservativeMFNbNlPvQcZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44BFC7: _D4core8internal2gc4impl12conservativeQw3Gcx__T7markAllS_DQCeQCcQBwQBwQBuQCfQBk16markConservativeMFNbNlPvQcZvZQClMFNbbZ14__foreachbody3MFNbKSQFjQEy1 gcinterface5RangeZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44FFB3: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi7opApplyMFNbMDFNbKQBwZiZ9__lambda2M NbKxSQEhQCsQCsQCiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x450359: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi13opApplyHelperFNbxPSQDnQDlQDfQCy__TQCvTQCsZQDd4NodeMDFN KxSQFaQDlQDlQDbZiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44FF81: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi7opAp lyMFNbMDFNbKQBwZiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44BF60: _D4core8internal2gc4impl12conservativeQw3Gcx__T7markAllS_DQCeQCcQBwQBwQBuQCfQBk16markConservativeMFN NlPvQcZvZQClMFNbbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x446752: _D4core8internal2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44B017: _D4core8internal2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCsQCqQCkQCkQCiQCtQBy18fullCollectNoStackMFNbZ2goFNbPSQEuQEsQEmQEmQEkQEv3Gcx mTQBbZQDsMFNbKQBnZm (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x443397: _D4core8internal2gc4impl12conservativeQw14ConservativeGC18full ollectNoStackMFNbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44332D: _D4core8internal2gc4impl12conservativeQw14ConservativeGC14collectNoStackMFNbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x4177B6: gc_term (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== ==185906== ==185906== HEAP SUMMARY: ==185906== in use at exit: 66,568 bytes in 6 blocks ==185906== total heap usage: 454 allocs, 448 frees, 346,252 bytes allocated ==185906== ==185906== 32 bytes in 1 blocks are possibly lost in loss record 2 of 6 ==185906== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==185906== by 0x441335: _D4core8internal2gc4impl12conservativeQw10initializeFZCQCbQBq11gcinterface2GC (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x43F608: _D4core2gc8registry16createGCInstanceFAyaZCQBpQBn11gcinterface2GC (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x43A2CA: gc_init_nothrow (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x417C93: _D4core8internal2gc4impl5protoQo7ProtoGC6qallocMFNbmkxC8TypeInfoZ QCm6memory8BlkInfo_ (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3DD216: gc_qalloc (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3E33FF: _d_newitemT (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F66B: _D3std5array__T8AppenderTAAyaZQp6__ctorMFNaNbNcNeQyZSQBzQBy__TQBvTQBpZQCd (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F564: _D3std5array__T8appenderTAAyaZQpFNaNbNfZSQBnQBm__T8AppenderTQBjZQo (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F4E8: _D3std5array__TQjTSQr9algorithm9iteration__T8splitterVAyaa6_61203d3d2062VEQCu8typecons__T4FlagVQBpa14_6b656570536570617261746f7273ZQBqi0TQDfTQDjZQDxFQDrQDuZ6Resul ZQGcFNaNbNfQGaZAQEv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F4CD: _D3std5array__T5splitTAyaTQeZQoFNaNbNfQqQsZAQw (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3EC601: _D3std6getopt11splitAndGetFNaNbNeAyaZSQBkQBj6Option (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== ==185906== 65,536 bytes in 1 blocks are definitely lost in loss record 6 of 6 ==185906== at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==185906== by 0x4206AA: _D4core6memory__T11pureReallocZQoFNaNbNiPvmZQe (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3FB799: _D3std12experimental9allocator10mallocator10Mallocator10reall cateMOFNaNbNiKAvmZb (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3A611B: _D3std12experimental9allocator__T11expandArrayTiTOSQBxQBwQBl10mallocator10MallocatorZQ aFNaNbNiKOQBvKAimZb (package.d:2143) ==185906== by 0x3A6024: _D4ddiv4core6memory9allocator__T11expandArrayTiTOS3std12experimentalQBx10mallocator10MallocatorZQCm NbNiKOQCfKAimAyaiZb (allocator.d:178) ==185906== by 0x3A04A7: _D4ddiv9container5stack__T11SimpleStackTiTS3std12experimental9allocator10mallocator10MallocatorZQ s6expandMFNbNiNemZv (stack.d:84) ==185906== by 0x3A0578: _D4ddiv9container5stack__T11SimpleStackTiTS3std12experimental9allocator10mallocator10Mallocator QCs4pushMFNbNiNfiZv (stack.d:105) ==185906== by 0x39FAE9: _D4ddiv9container5stack18__unittest_L142_C1FNfZv (stack.d:166) ==185906== by 0x3BD004: _D5silly11executeTestFSQv4TestZSQBe10TestResult (silly.d:189) ==185906== by 0x3BC89B: _D5silly24_sharedStaticCtor_L24_C1FZ9__lambda1FZ15__foreachbody15MFKSQCp4TestZi (silly.d:109) ==185906== by 0x3C2614: _D3std11parallelism__T14doSizeZeroCaseTAS5silly4TestTDFKQqZiZQBnFKSQCnQCm__T15Paralle ForeachTQCdZQwQBvZi (parallelism.d:3748) ==185906== by 0x3C2013: _D3std11parallelism__T15ParallelForeachTAS5silly4TestZQBg7opApplyMFMDFKQBeZiZi (parallelism.d-mixin-4039:4043) ==185906== ==185906== LEAK SUMMARY: ==185906== definitely lost: 65,536 bytes in 1 blocks ==185906== indirectly lost: 0 bytes in 0 blocks ==185906== possibly lost: 32 bytes in 1 blocks ==185906== still reachable: 1,000 bytes in 4 blocks ==185906== suppressed: 0 bytes in 0 blocks ==185906== Reachable blocks (those to which a pointer was found) are not shown. ==185906== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==185906== ==185906== Use --track-origins=yes to see where uninitialised values come from ==185906== For lists of detected and suppressed errors, rerun with: -s ==185906== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0) ```
Jun 20
parent Basile B. <b2.temp gmx.com> writes:
On Sunday, 20 June 2021 at 17:58:41 UTC, Luis wrote:
 On Sunday, 20 June 2021 at 16:53:06 UTC, Basile B. wrote:
 On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:
Yeah. I know Valgrind. I used it a lot, when I did C or C++ stuff, and it's a wonderful tool. However have his problems : 1. Makes anything to run painfull slow.
alas nothing can be done against this to my knowledge.
 2. At least with DMD, the mangled information & stack traces 
 are hard to follow.
here you can definitively use ddemangle.
Jun 20
prev sibling next sibling parent reply max haughton <maxhaton gmail.com> writes:
On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:
 Well, this last week I was playing using malloc/free to do some 
  nogc code, and using the std.experimental.allocator API. Also, 
 I was trying to see a way to profile my code and search for 
 memory leaks, etc...

 [...]
You aren't using address sanitizer?
Jun 20
parent reply Luis <Luis.panadero gmail.com> writes:
On Sunday, 20 June 2021 at 17:35:23 UTC, max haughton wrote:
 On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:
 Well, this last week I was playing using malloc/free to do 
 some  nogc code, and using the std.experimental.allocator API. 
 Also, I was trying to see a way to profile my code and search 
 for memory leaks, etc...

 [...]
You aren't using address sanitizer?
I'm using DMD, not LDC or GDC.
Jun 20
parent max haughton <maxhaton gmail.com> writes:
On Sunday, 20 June 2021 at 17:42:49 UTC, Luis wrote:
 On Sunday, 20 June 2021 at 17:35:23 UTC, max haughton wrote:
 On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:
 Well, this last week I was playing using malloc/free to do 
 some  nogc code, and using the std.experimental.allocator 
 API. Also, I was trying to see a way to profile my code and 
 search for memory leaks, etc...

 [...]
You aren't using address sanitizer?
I'm using DMD, not LDC or GDC.
You should set up a way to use LDC then, ASan is too good not to use.
Jun 20
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/21 11:49 AM, Luis wrote:
 Well, this last week I was playing using malloc/free to do some  nogc 
 code, and using the std.experimental.allocator API. Also, I was trying 
 to see a way to profile my code and search for memory leaks, etc...
 
 What I initially did, was implementing my own allocator API plus my own 
 mallocator, seeing how ECS buble engine does this.
 Later, I take a look how std.experimental.allocator API works and I 
 ended writing my own wrapper above make, makeArray, dispose, expandArray 
 & shrinkArray that implements this profiler/stats, and using 
 std.experimental.mallocator .
 Finally, I catch that I can only use my wrapper on my code, and that I 
 couldn't use on other third party libraries that allow to use allocators 
 (like EMSI Containers), because (obviously) uses the 
 std.experimental.allocator API.
 Then, I discover 
 std.experimental.allocator.building_blocks.stats_collector that looks 
 that does what my littler profiler does, and more. But I see tiny detail 
 that I don't see totally ok.
 
 The stats that are reported by reportPerCallStatistics, shows the file & 
 line number of the allocator!
 
 Like this :
 
 ```
 /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): 
 [numAllocate:1, numAllocateOK:1, bytesAllocated:40]
 ```
 
 This could be useful for someone writing his own allocator, but it's 
 pretty useless for anyone trying to track where a memory leak originated 
 in his own code!
 
 On my wrapper, What I did was to have a `string file = __FILE__, int 
 line = __LINE__` parameters append to the make, makeArray, dispose, 
 etc... functions and uses the file and & line to record where something 
 was allocated/deallocated/reallocated.
 
 So, why std.experimental.allocator API not does something similar to 
 this ? The allocator implementations, would need to pass & ignore these 
 two additional parameters. Except StatsCollector that can use it to show 
 where the allocation/deallocation/reallocation it's happening
If I understand the problem correctly, in order for statsCollector to work, it needs to be at the "top" of the allocator type tree. The design indeed does not support line numbers when statsAllcator is inserted somewhere in the middle.
Jun 21