digitalmars.D - std.experimental.allocator API and nearly useless StatsCollector
- Luis (36/36) Jun 20 2021 Well, this last week I was playing using malloc/free to do some
- Basile B. (6/20) Jun 20 2021 To understand leaks I'd use valgrind instead. As it is based on
- Luis (154/179) Jun 20 2021 Yeah. I know Valgrind. I used it a lot, when I did C or C++
- Basile B. (3/10) Jun 20 2021 here you can definitively use ddemangle.
- max haughton (2/7) Jun 20 2021 You aren't using address sanitizer?
- Luis (2/10) Jun 20 2021 I'm using DMD, not LDC or GDC.
- max haughton (3/14) Jun 20 2021 You should set up a way to use LDC then, ASan is too good not to
- Andrei Alexandrescu (5/47) Jun 21 2021 If I understand the problem correctly, in order for statsCollector to
Well, this last week I was playing using malloc/free to do some nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc... What I initially did, was implementing my own allocator API plus my own mallocator, seeing how ECS buble engine does this. Later, I take a look how std.experimental.allocator API works and I ended writing my own wrapper above make, makeArray, dispose, expandArray & shrinkArray that implements this profiler/stats, and using std.experimental.mallocator . Finally, I catch that I can only use my wrapper on my code, and that I couldn't use on other third party libraries that allow to use allocators (like EMSI Containers), because (obviously) uses the std.experimental.allocator API. Then, I discover std.experimental.allocator.building_blocks.stats_collector that looks that does what my littler profiler does, and more. But I see tiny detail that I don't see totally ok. The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator! Like this : ``` /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40] ``` This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code! On my wrapper, What I did was to have a `string file = __FILE__, int line = __LINE__` parameters append to the make, makeArray, dispose, etc... functions and uses the file and & line to record where something was allocated/deallocated/reallocated. So, why std.experimental.allocator API not does something similar to this ? The allocator implementations, would need to pass & ignore these two additional parameters. Except StatsCollector that can use it to show where the allocation/deallocation/reallocation it's happening
Jun 20 2021
On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:Well, this last week I was playing using malloc/free to do some nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc... [...] The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator! Like this : ``` /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40] ``` This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code!To understand leaks I'd use valgrind instead. As it is based on DWARF info, reported leaking blocks are accompanied with a pretty (after demangling) back trace. So the first entry for each block will be unsurprinsingly always the same (e.g glibc malloc) but the other can be more interesting.
Jun 20 2021
On Sunday, 20 June 2021 at 16:53:06 UTC, Basile B. wrote:On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:Yeah. I know Valgrind. I used it a lot, when I did C or C++ stuff, and it's a wonderful tool. However have his problems : 1. Makes anything to run painfull slow. 2. At least with DMD, the mangled information & stack traces are hard to follow. Compare this two outputs when I comment the code that calls to allocator.dispose(): My simple & dirty profiler : ``` $ ./bin/ddiv-test-unittest -t 1 -i "Stack" ✓ ddiv.container.stack SimpleStack Summary: 1 passed, 0 failed in 0 ms WARNING! Possible memory leaks! ----- Memory allocation information ----- Total amount of allocated memory that has not been released: 65536 bytes Addr: 0x55C70F4D0E60 - [int[]] 65536 bytes - source/ddiv/container/stack.d:84 ``` Valgrind (fun thing. I need to use --leak-check=full to see the memory leak of my code, at at 0x483DFAF realloc. The rest looks that it's from Silly and/or D runtime): ``` $ valgrind --leak-check=full ./bin/ddiv-test-unittest -t 1 -i "Stack" ==185906== Memcheck, a memory error detector ==185906== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==185906== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info ==185906== Command: ./bin/ddiv-test-unittest -t 1 -i Stack ==185906== --185906-- WARNING: Serious error when reading debug info --185906-- When reading debug info from /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest: --185906-- DWARF line info appears to be corrupt - the section is too small --185906-- WARNING: Serious error when reading debug info --185906-- When reading debug info from /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest: --185906-- read_filename_table: .debug_line is missing? ✓ ddiv.container.stack SimpleStack Summary: 1 passed, 0 failed in 89 ms ==185906== Conditional jump or move depends on uninitialised value(s) ==185906== at 0x44B2C8: _D4core8internal2gc4impl12conservativeQw3Gcx__T4markVbi0Vbi0ZQoMFNbNlSQCqQCoQCiQCiQCgQCrQBw__ 9ScanRangeVbi0ZQpZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x445469: _D4core8internal2gc4impl12conservativeQw3Gcx16markConservativeMFNbNlPvQcZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44BFC7: _D4core8internal2gc4impl12conservativeQw3Gcx__T7markAllS_DQCeQCcQBwQBwQBuQCfQBk16markConservativeMFNbNlPvQcZvZQClMFNbbZ14__foreachbody3MFNbKSQFjQEy1 gcinterface5RangeZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44FFB3: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi7opApplyMFNbMDFNbKQBwZiZ9__lambda2M NbKxSQEhQCsQCsQCiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x450359: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi13opApplyHelperFNbxPSQDnQDlQDfQCy__TQCvTQCsZQDd4NodeMDFN KxSQFaQDlQDlQDbZiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44FF81: _D4core8internal9container5treap__T5TreapTSQBp2gc11gcinterface5RangeZQBi7opAp lyMFNbMDFNbKQBwZiZi (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44BF60: _D4core8internal2gc4impl12conservativeQw3Gcx__T7markAllS_DQCeQCcQBwQBwQBuQCfQBk16markConservativeMFN NlPvQcZvZQClMFNbbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x446752: _D4core8internal2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44B017: _D4core8internal2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCsQCqQCkQCkQCiQCtQBy18fullCollectNoStackMFNbZ2goFNbPSQEuQEsQEmQEmQEkQEv3Gcx mTQBbZQDsMFNbKQBnZm (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x443397: _D4core8internal2gc4impl12conservativeQw14ConservativeGC18full ollectNoStackMFNbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x44332D: _D4core8internal2gc4impl12conservativeQw14ConservativeGC14collectNoStackMFNbZv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x4177B6: gc_term (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== ==185906== ==185906== HEAP SUMMARY: ==185906== in use at exit: 66,568 bytes in 6 blocks ==185906== total heap usage: 454 allocs, 448 frees, 346,252 bytes allocated ==185906== ==185906== 32 bytes in 1 blocks are possibly lost in loss record 2 of 6 ==185906== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==185906== by 0x441335: _D4core8internal2gc4impl12conservativeQw10initializeFZCQCbQBq11gcinterface2GC (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x43F608: _D4core2gc8registry16createGCInstanceFAyaZCQBpQBn11gcinterface2GC (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x43A2CA: gc_init_nothrow (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x417C93: _D4core8internal2gc4impl5protoQo7ProtoGC6qallocMFNbmkxC8TypeInfoZ QCm6memory8BlkInfo_ (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3DD216: gc_qalloc (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3E33FF: _d_newitemT (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F66B: _D3std5array__T8AppenderTAAyaZQp6__ctorMFNaNbNcNeQyZSQBzQBy__TQBvTQBpZQCd (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F564: _D3std5array__T8appenderTAAyaZQpFNaNbNfZSQBnQBm__T8AppenderTQBjZQo (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F4E8: _D3std5array__TQjTSQr9algorithm9iteration__T8splitterVAyaa6_61203d3d2062VEQCu8typecons__T4FlagVQBpa14_6b656570536570617261746f7273ZQBqi0TQDfTQDjZQDxFQDrQDuZ6Resul ZQGcFNaNbNfQGaZAQEv (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x41F4CD: _D3std5array__T5splitTAyaTQeZQoFNaNbNfQqQsZAQw (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3EC601: _D3std6getopt11splitAndGetFNaNbNeAyaZSQBkQBj6Option (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== ==185906== 65,536 bytes in 1 blocks are definitely lost in loss record 6 of 6 ==185906== at 0x483DFAF: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==185906== by 0x4206AA: _D4core6memory__T11pureReallocZQoFNaNbNiPvmZQe (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3FB799: _D3std12experimental9allocator10mallocator10Mallocator10reall cateMOFNaNbNiKAvmZb (in /home/luis/repos/dlang/ddiv/bin/ddiv-test-unittest) ==185906== by 0x3A611B: _D3std12experimental9allocator__T11expandArrayTiTOSQBxQBwQBl10mallocator10MallocatorZQ aFNaNbNiKOQBvKAimZb (package.d:2143) ==185906== by 0x3A6024: _D4ddiv4core6memory9allocator__T11expandArrayTiTOS3std12experimentalQBx10mallocator10MallocatorZQCm NbNiKOQCfKAimAyaiZb (allocator.d:178) ==185906== by 0x3A04A7: _D4ddiv9container5stack__T11SimpleStackTiTS3std12experimental9allocator10mallocator10MallocatorZQ s6expandMFNbNiNemZv (stack.d:84) ==185906== by 0x3A0578: _D4ddiv9container5stack__T11SimpleStackTiTS3std12experimental9allocator10mallocator10Mallocator QCs4pushMFNbNiNfiZv (stack.d:105) ==185906== by 0x39FAE9: _D4ddiv9container5stack18__unittest_L142_C1FNfZv (stack.d:166) ==185906== by 0x3BD004: _D5silly11executeTestFSQv4TestZSQBe10TestResult (silly.d:189) ==185906== by 0x3BC89B: _D5silly24_sharedStaticCtor_L24_C1FZ9__lambda1FZ15__foreachbody15MFKSQCp4TestZi (silly.d:109) ==185906== by 0x3C2614: _D3std11parallelism__T14doSizeZeroCaseTAS5silly4TestTDFKQqZiZQBnFKSQCnQCm__T15Paralle ForeachTQCdZQwQBvZi (parallelism.d:3748) ==185906== by 0x3C2013: _D3std11parallelism__T15ParallelForeachTAS5silly4TestZQBg7opApplyMFMDFKQBeZiZi (parallelism.d-mixin-4039:4043) ==185906== ==185906== LEAK SUMMARY: ==185906== definitely lost: 65,536 bytes in 1 blocks ==185906== indirectly lost: 0 bytes in 0 blocks ==185906== possibly lost: 32 bytes in 1 blocks ==185906== still reachable: 1,000 bytes in 4 blocks ==185906== suppressed: 0 bytes in 0 blocks ==185906== Reachable blocks (those to which a pointer was found) are not shown. ==185906== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==185906== ==185906== Use --track-origins=yes to see where uninitialised values come from ==185906== For lists of detected and suppressed errors, rerun with: -s ==185906== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0) ```Well, this last week I was playing using malloc/free to do some nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc... [...] The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator! Like this : ``` /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40] ``` This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code!To understand leaks I'd use valgrind instead. As it is based on DWARF info, reported leaking blocks are accompanied with a pretty (after demangling) back trace. So the first entry for each block will be unsurprinsingly always the same (e.g glibc malloc) but the other can be more interesting.
Jun 20 2021
Basile B. <b2.temp gmx.com> On Sunday, 20 June 2021 at 17:58:41 UTC, Luis wrote:On Sunday, 20 June 2021 at 16:53:06 UTC, Basile B. wrote:alas nothing can be done against this to my knowledge.On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:Yeah. I know Valgrind. I used it a lot, when I did C or C++ stuff, and it's a wonderful tool. However have his problems : 1. Makes anything to run painfull slow.2. At least with DMD, the mangled information & stack traces are hard to follow.here you can definitively use ddemangle.
Jun 20 2021
On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:Well, this last week I was playing using malloc/free to do some nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc... [...]You aren't using address sanitizer?
Jun 20 2021
On Sunday, 20 June 2021 at 17:35:23 UTC, max haughton wrote:On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:I'm using DMD, not LDC or GDC.Well, this last week I was playing using malloc/free to do some nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc... [...]You aren't using address sanitizer?
Jun 20 2021
On Sunday, 20 June 2021 at 17:42:49 UTC, Luis wrote:On Sunday, 20 June 2021 at 17:35:23 UTC, max haughton wrote:You should set up a way to use LDC then, ASan is too good not to use.On Sunday, 20 June 2021 at 15:49:13 UTC, Luis wrote:I'm using DMD, not LDC or GDC.Well, this last week I was playing using malloc/free to do some nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc... [...]You aren't using address sanitizer?
Jun 20 2021
On 6/20/21 11:49 AM, Luis wrote:Well, this last week I was playing using malloc/free to do some nogc code, and using the std.experimental.allocator API. Also, I was trying to see a way to profile my code and search for memory leaks, etc... What I initially did, was implementing my own allocator API plus my own mallocator, seeing how ECS buble engine does this. Later, I take a look how std.experimental.allocator API works and I ended writing my own wrapper above make, makeArray, dispose, expandArray & shrinkArray that implements this profiler/stats, and using std.experimental.mallocator . Finally, I catch that I can only use my wrapper on my code, and that I couldn't use on other third party libraries that allow to use allocators (like EMSI Containers), because (obviously) uses the std.experimental.allocator API. Then, I discover std.experimental.allocator.building_blocks.stats_collector that looks that does what my littler profiler does, and more. But I see tiny detail that I don't see totally ok. The stats that are reported by reportPerCallStatistics, shows the file & line number of the allocator! Like this : ``` /usr/include/dmd/phobos/std/experimental/allocator/package.d(1585): [numAllocate:1, numAllocateOK:1, bytesAllocated:40] ``` This could be useful for someone writing his own allocator, but it's pretty useless for anyone trying to track where a memory leak originated in his own code! On my wrapper, What I did was to have a `string file = __FILE__, int line = __LINE__` parameters append to the make, makeArray, dispose, etc... functions and uses the file and & line to record where something was allocated/deallocated/reallocated. So, why std.experimental.allocator API not does something similar to this ? The allocator implementations, would need to pass & ignore these two additional parameters. Except StatsCollector that can use it to show where the allocation/deallocation/reallocation it's happeningIf I understand the problem correctly, in order for statsCollector to work, it needs to be at the "top" of the allocator type tree. The design indeed does not support line numbers when statsAllcator is inserted somewhere in the middle.
Jun 21 2021