www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - malloc error when trying to assign the returned pointer to a struct

reply rempas <rempas tutanota.com> writes:
I do have the following struct:

```d
struct Vec(T) {
private:
   T* _ptr = null; // The pointer to the data
   u64 _cap = 0;   // Total amount of elements (not bytes) we can 
store

public:
   /* Create a vector by just allocating memory for it. The null 
terminator is not set for
      strings as, the vector is considered empty and we should  
first push something to it
      in order to use it! */
   this(i64 size) {
     this._len = 0;
     this._cap = size;

     static if (is(T == char)) { size += 1; } // Additional space 
for the null terminator
     this._ptr = cast(T*)malloc(size);
   }
}
```

That's some minimal code that I do have just to showcase it. So, 
some times, this work will works, some others, it will give me 
the following error:

`Fatal glibc error: malloc.c:2594 (sysmalloc): assertion failed: 
(old_top == initial_top (av) && old_size == 0) || ((unsigned 
long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned 
long) old_end & (pagesize - 1)) == 0)`

The problem seems to happen when the pointer that is returned 
from `malloc` is assigned to the `_ptr` field. If I just assign 
it to a variable and don't assign anything to `_ptr`, it will 
work!

Is there any possible that there is a compiler bug? I do use ldc2 
and `betterC`!
Sep 08 2023
next sibling parent reply evilrat <evilrat66 gmail.com> writes:
On Friday, 8 September 2023 at 07:59:37 UTC, rempas wrote:

 I do have the following struct:

 ```d
 struct Vec(T) {
 private:
   T* _ptr = null; // The pointer to the data
   u64 _cap = 0;   // Total amount of elements (not bytes) we 
 can store

 public:
   /* Create a vector by just allocating memory for it. The null 
 terminator is not set for
      strings as, the vector is considered empty and we should  
 first push something to it
      in order to use it! */
   this(i64 size) {
     this._len = 0;
     this._cap = size;

     static if (is(T == char)) { size += 1; } // Additional 
 space for the null terminator
     this._ptr = cast(T*)malloc(size);
   }
 }
 ```

 That's some minimal code that I do have just to showcase it. 
 So, some times, this work will works, some others, it will give 
 me the following error:

 `Fatal glibc error: malloc.c:2594 (sysmalloc): assertion 
 failed: (old_top == initial_top (av) && old_size == 0) || 
 ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) 
 && ((unsigned long) old_end & (pagesize - 1)) == 0)`

 The problem seems to happen when the pointer that is returned 
 from `malloc` is assigned to the `_ptr` field. If I just assign 
 it to a variable and don't assign anything to `_ptr`, it will 
 work!

 Is there any possible that there is a compiler bug? I do use 
 ldc2 and `betterC`!


Hard to tell from that code but it is quite unlikely there is a 
compiler bug in such simple use case.

I assume you already tried debugging your program, right?

So how about to diagnose a bit more, what if you enforce check 
before malloc that size>0, and second - from that example it is 
unclear how you are using that struct, so maybe add else 
statement static assert to see if it is misused somewhere else in 
your codebase?

Also this example doesn't have len field, depending on how you 
use with regard to cap this could be a source of problems too.
Sep 08 2023
parent reply rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 09:07:12 UTC, evilrat wrote:

 Hard to tell from that code but it is quite unlikely there is a 
 compiler bug in such simple use case.

 I assume you already tried debugging your program, right?


Yep! I have spent days and it's these kinds of bugs that burn me 
off and make me want to give up.


 So how about to diagnose a bit more, what if you enforce check 
 before malloc that size>0,


In the real code, there is a check! I just removed it from this 
example to make the code more clear.


 and second - from that example it is unclear how you are using 
 that struct, so maybe add else statement static assert to see 
 if it is misused somewhere else in your codebase?


That's interesting, I wasn't able to find something else! The bug 
happens when I run the testing suit and well... the tests before 
pass so I cannot find anything that goes wrong except for the 
fact that I do not free the memory that is allocated (on purpose).

But still, why would the bug be there when I assign the `_ptr` on 
the field and it won't happen if I don't assign it and just use a 
regular local variable instead (and don't assign anything to the 
`_ptr` field)?


 Also this example doesn't have len field, depending on how you 
 use with regard to cap this could be a source of problems too.


Oh right, I forgot about that! The `_len` field normally exists 
and it's it's also an unsigned 64-bit (u64) number! Yeah, the 
code is a little bit changed from the original to make it more 
simple for you guys...
Sep 08 2023
parent reply evilrat <evilrat666 gmail.com> writes:
On Friday, 8 September 2023 at 11:50:52 UTC, rempas wrote:

 That's interesting, I wasn't able to find something else! The 
 bug happens when I run the testing suit and well... the tests 
 before pass so I cannot find anything that goes wrong except 
 for the fact that I do not free the memory that is allocated 
 (on purpose).


You run with -unittest compiler flag? Well, that does nothing for 
me with betterc (without it is ok).

I did stupid and unsafe things like malloc(0) and writing out of 
bounds but still no crash, it works fine.

I guess it depends on your libc, tested this on ubuntu 23.04 with 
gcc12 install and ldc 1.32.2

```d
import core.stdc.stdlib;
import core.stdc.stdio;

alias u64 = ulong;
alias i64 = long;

struct Vec(T) {
private:
   T* _ptr = null; // The pointer to the data
   u64 _cap = 0;   // Total amount of elements (not bytes) we can 
store
   u64 _len = 0;

public:
   /* Create a vector by just allocating memory for it. The null 
terminator is not set for
      strings as, the vector is considered empty and we should  
first push something to it
      in order to use it! */
   this(i64 size) {
     this._len = 0;
     this._cap = size;

     static if (is(T == char)) { size += 1; } // Additional space 
for the null terminator
     this._ptr = cast(T*)malloc(size);
   }

   ref T opIndex(size_t idx) { return _ptr[idx]; }
}

extern(C)
void main()
//unittest
{
     enum el = 3;
     auto vec = Vec!char(10);
     assert(vec._ptr);
     vec[el] = 'h';
     assert(vec[el] == 'h');
     printf("ptr = %p\n", vec._ptr);
     printf("vec ptr = %p\n", &vec[el]);
     printf("vec local = %p\n", &vec);
     printf("vec[%d] = %c\n", el, vec[el]);
     foreach (i; 0..vec._cap) {
       printf("-");
     }
     printf("\n");
     foreach (i; 0..vec._cap) {
       printf("%d", vec[i]);
     }
     printf("\n");
     printf("run ok\n");
}
```

ldc2 -betterC -run membug.d

output

```
ptr = 0x55cb701de2a0
vec ptr = 0x55cb701de2a3
vec local = 0x7fffa1542258
vec[3] = h
----------
000104000000
run ok
```
Sep 08 2023
parent reply rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 13:05:47 UTC, evilrat wrote:

 You run with -unittest compiler flag? Well, that does nothing 
 for me with betterc (without it is ok).

 I did stupid and unsafe things like malloc(0) and writing out 
 of bounds but still no crash, it works fine.

 I guess it depends on your libc, tested this on ubuntu 23.04 
 with gcc12 install and ldc 1.32.2

 ```d
 import core.stdc.stdlib;
 import core.stdc.stdio;

 alias u64 = ulong;
 alias i64 = long;

 struct Vec(T) {
 private:
   T* _ptr = null; // The pointer to the data
   u64 _cap = 0;   // Total amount of elements (not bytes) we 
 can store
   u64 _len = 0;

 public:
   /* Create a vector by just allocating memory for it. The null 
 terminator is not set for
      strings as, the vector is considered empty and we should  
 first push something to it
      in order to use it! */
   this(i64 size) {
     this._len = 0;
     this._cap = size;

     static if (is(T == char)) { size += 1; } // Additional 
 space for the null terminator
     this._ptr = cast(T*)malloc(size);
   }

   ref T opIndex(size_t idx) { return _ptr[idx]; }
 }

 extern(C)
 void main()
 //unittest
 {
     enum el = 3;
     auto vec = Vec!char(10);
     assert(vec._ptr);
     vec[el] = 'h';
     assert(vec[el] == 'h');
     printf("ptr = %p\n", vec._ptr);
     printf("vec ptr = %p\n", &vec[el]);
     printf("vec local = %p\n", &vec);
     printf("vec[%d] = %c\n", el, vec[el]);
     foreach (i; 0..vec._cap) {
       printf("-");
     }
     printf("\n");
     foreach (i; 0..vec._cap) {
       printf("%d", vec[i]);
     }
     printf("\n");
     printf("run ok\n");
 }
 ```

 ldc2 -betterC -run membug.d

 output

 ```
 ptr = 0x55cb701de2a0
 vec ptr = 0x55cb701de2a3
 vec local = 0x7fffa1542258
 vec[3] = h
 ----------
 000104000000
 run ok
 ```


As D's "uninttest" feature is disabled on BetterC, I have wrote 
my own testing suit (with is very simple). I just said that to 
point out that I'm testing this data structure (along side other 
things) and I cannot find anything wrong.

I have made a search on the web and I found out one thread that 
pointed out that it may be a Glibc error. However, because like I 
said the problem only happens when I assign the returned value to 
the `_ptr` field, I just wanted to post here in case someone has 
a similar experience and if it's a compiler bug in which case, we 
should report it.

I will use a VM to test my code in another environment and I'll 
be back to report!
Sep 08 2023
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
In case you didn't know, all you need to get unittests working in 
-betterC is:

```d
			foreach (module_; allModules) {
				foreach (unitTest; __traits(getUnitTests, module_)) {
					unitTest();
				}
			}
```

You'd need to provide allModules somehow, like dub does.

https://github.com/dlang/dub/blob/2ea883833adf085095b07a7dba8250fb3db79a71/source/dub/project.d#L1937
Sep 08 2023
next sibling parent reply rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 13:34:42 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 In case you didn't know, all you need to get unittests working 
 in -betterC is:

 ```d
 			foreach (module_; allModules) {
 				foreach (unitTest; __traits(getUnitTests, module_)) {
 					unitTest();
 				}
 			}
 ```

 You'd need to provide allModules somehow, like dub does.

 https://github.com/dlang/dub/blob/2ea883833adf085095b07a7dba8250fb3db79a71/source/dub/project.d#L1937


Oh wow!!! How did I missed that? It's either recent (past months) 
or I'm blind, lol!

As I don't remember unittests however, I would like to as. Do 
they have automatic symbol order resolution? Which is, testing 
symbols that other symbol depend on first? Or is it random?
Sep 08 2023
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 09/09/2023 2:20 AM, rempas wrote:

 Do they have automatic symbol order resolution? Which is, testing 
 symbols that other symbol depend on first? Or is it random?


No, for this you need ModuleInfo. The order is sequential on what it 
sees first.

Personally I test using full D rather than -betterC.

For dub:

```json
     "configurations": [
         {
             "name": "library",
             "targetType": "dynamicLibrary",
             "versions": [
                 "DynamicSideroBase"
             ],
             "buildOptions": [
                 "betterC"
             ]
         },
         {
             "name": "static",
             "targetType": "staticLibrary",
             "buildOptions": [
                 "betterC"
             ]
         },
         {
             "name": "unittest"
         },
         {
             "name": "executable",
             "targetType": "executable"
         }
     ]
```
Sep 08 2023
parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 14:40:13 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 No, for this you need ModuleInfo. The order is sequential on 
 what it sees first.

 Personally I test using full D rather than -betterC.

 For dub:

 ```json
     "configurations": [
         {
             "name": "library",
             "targetType": "dynamicLibrary",
             "versions": [
                 "DynamicSideroBase"
             ],
             "buildOptions": [
                 "betterC"
             ]
         },
         {
             "name": "static",
             "targetType": "staticLibrary",
             "buildOptions": [
                 "betterC"
             ]
         },
         {
             "name": "unittest"
         },
         {
             "name": "executable",
             "targetType": "executable"
         }
     ]
 ```


Thank you for all the information! I really appreciate it my 
friend!
Sep 08 2023
prev sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Friday, 8 September 2023 at 13:34:42 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 In case you didn't know, all you need to get unittests working 
 in -betterC is:

 ```d
 			foreach (module_; allModules) {
 				foreach (unitTest; __traits(getUnitTests, module_)) {
 					unitTest();
 				}
 			}
 ```

 You'd need to provide allModules somehow, like dub does.

 https://github.com/dlang/dub/blob/2ea883833adf085095b07a7dba8250fb3db79a71/source/dub/project.d#L1937


TIL, thanks for sharing, this will be very useful!
Sep 10 2023
prev sibling parent reply Kagamin <spam here.lot> writes:
On Friday, 8 September 2023 at 13:32:00 UTC, rempas wrote:

 On Friday, 8 September 2023 at 13:05:47 UTC, evilrat wrote:

 ```d
 import core.stdc.stdlib;
 import core.stdc.stdio;

 alias u64 = ulong;
 alias i64 = long;

 struct Vec(T) {
 private:
   T* _ptr = null; // The pointer to the data
   u64 _cap = 0;   // Total amount of elements (not bytes) we 
 can store
   u64 _len = 0;

 public:
   /* Create a vector by just allocating memory for it. The 
 null terminator is not set for
      strings as, the vector is considered empty and we should  
 first push something to it
      in order to use it! */
   this(i64 size) {
     this._len = 0;
     this._cap = size;

     static if (is(T == char)) { size += 1; } // Additional 
 space for the null terminator
     this._ptr = cast(T*)malloc(size);
   }

   ref T opIndex(size_t idx) { return _ptr[idx]; }
 }

 extern(C)
 void main()
 //unittest
 {
     enum el = 3;
     auto vec = Vec!char(10);
     assert(vec._ptr);
     vec[el] = 'h';
     assert(vec[el] == 'h');
     printf("ptr = %p\n", vec._ptr);
     printf("vec ptr = %p\n", &vec[el]);
     printf("vec local = %p\n", &vec);
     printf("vec[%d] = %c\n", el, vec[el]);
     foreach (i; 0..vec._cap) {
       printf("-");
     }
     printf("\n");
     foreach (i; 0..vec._cap) {
       printf("%d", vec[i]);
     }
     printf("\n");
     printf("run ok\n");
 }
 ```

 ldc2 -betterC -run membug.d

 output

 ```
 ptr = 0x55cb701de2a0
 vec ptr = 0x55cb701de2a3
 vec local = 0x7fffa1542258
 vec[3] = h
 ----------
 000104000000
 run ok
 ```


 I have made a search on the web and I found out one thread that 
 pointed out that it may be a Glibc error. However, because like 
 I said the problem only happens when I assign the returned 
 value to the `_ptr` field, I just wanted to post here in case 
 someone has a similar experience and if it's a compiler bug in 
 which case, we should report it.


Did you run this example program above? Does it crash?
Sep 08 2023
parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 14:50:17 UTC, Kagamin wrote:

 Did you run this example program above? Does it crash?


I didn't as I suppose that it would had no problems as it works 
for you. Either that, or it will be indeed a problem with my 
Glibc.

I did run it now after your reply and it works without any 
problems!
Sep 08 2023
prev sibling next sibling parent reply Hipreme <msnmancini hotmail.com> writes:
On Friday, 8 September 2023 at 07:59:37 UTC, rempas wrote:

 I do have the following struct:

 ```d
 struct Vec(T) {
 private:
   T* _ptr = null; // The pointer to the data
   u64 _cap = 0;   // Total amount of elements (not bytes) we 
 can store

 public:
   /* Create a vector by just allocating memory for it. The null 
 terminator is not set for
      strings as, the vector is considered empty and we should  
 first push something to it
      in order to use it! */
   this(i64 size) {
     this._len = 0;
     this._cap = size;

     static if (is(T == char)) { size += 1; } // Additional 
 space for the null terminator
     this._ptr = cast(T*)malloc(size);
   }
 }
 ```

 That's some minimal code that I do have just to showcase it. 
 So, some times, this work will works, some others, it will give 
 me the following error:

 `Fatal glibc error: malloc.c:2594 (sysmalloc): assertion 
 failed: (old_top == initial_top (av) && old_size == 0) || 
 ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) 
 && ((unsigned long) old_end & (pagesize - 1)) == 0)`

 The problem seems to happen when the pointer that is returned 
 from `malloc` is assigned to the `_ptr` field. If I just assign 
 it to a variable and don't assign anything to `_ptr`, it will 
 work!

 Is there any possible that there is a compiler bug? I do use 
 ldc2 and `betterC`!




Hello, not completely unrelated to your problem, I have also done 
something like that, and when you're in D, don't use simply a 
pointer and length like that, use the `slice` operator.

See references:

https://tour.dlang.org/tour/en/basics/slices
https://dlang.org/spec/operatoroverloading.html#array-ops


For example, you can make your pointer a lot safer by doing:

```d
size_t length = 5;
int* pointer = cast(int*)malloc(int.sizeof * length); //Don't
int[] mallocArray = (cast(int*)malloc(int.sizeof * 
length))[0..length]; //Do
```

On the second way, you'll get bound checks, thus, making it 
safer. Also, no need to keep track of your length separately 
anymore.

This is good practice in D language and you'll find yourself 
using this instead in the future.

And yes, this works in betterC, it is a simple runtime check, 
completely ` nogc  safe nothrow` and every other kind of thing 
you would want.
Sep 08 2023
parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 09:25:59 UTC, Hipreme wrote:

 Hello, not completely unrelated to your problem, I have also 
 done something like that, and when you're in D, don't use 
 simply a pointer and length like that, use the `slice` operator.

 See references:

 https://tour.dlang.org/tour/en/basics/slices
 https://dlang.org/spec/operatoroverloading.html#array-ops


 For example, you can make your pointer a lot safer by doing:

 ```d
 size_t length = 5;
 int* pointer = cast(int*)malloc(int.sizeof * length); //Don't
 int[] mallocArray = (cast(int*)malloc(int.sizeof * 
 length))[0..length]; //Do
 ```

 On the second way, you'll get bound checks, thus, making it 
 safer. Also, no need to keep track of your length separately 
 anymore.

 This is good practice in D language and you'll find yourself 
 using this instead in the future.

 And yes, this works in betterC, it is a simple runtime check, 
 completely ` nogc  safe nothrow` and every other kind of thing 
 you would want.


Thank you for the reply! Using slices will give me the same 
result (my code has another field that I forgot to mention btw 
and it's `_len` (which holds the length that is used from the 
capacity). However, this will just make the API worse to use so I 
choose to not use slices in this case as using separate fields is 
more convenient.
Sep 08 2023
prev sibling next sibling parent bachmeier <no spam.net> writes:
On Friday, 8 September 2023 at 07:59:37 UTC, rempas wrote:

 I do have the following struct:

 ```d
 struct Vec(T) {
 private:
   T* _ptr = null; // The pointer to the data
   u64 _cap = 0;   // Total amount of elements (not bytes) we 
 can store

 public:
   /* Create a vector by just allocating memory for it. The null 
 terminator is not set for
      strings as, the vector is considered empty and we should  
 first push something to it
      in order to use it! */
   this(i64 size) {
     this._len = 0;
     this._cap = size;

     static if (is(T == char)) { size += 1; } // Additional 
 space for the null terminator
     this._ptr = cast(T*)malloc(size);
   }
 }
 ```

 That's some minimal code that I do have just to showcase it. 
 So, some times, this work will works, some others, it will give 
 me the following error:

 `Fatal glibc error: malloc.c:2594 (sysmalloc): assertion 
 failed: (old_top == initial_top (av) && old_size == 0) || 
 ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) 
 && ((unsigned long) old_end & (pagesize - 1)) == 0)`

 The problem seems to happen when the pointer that is returned 
 from `malloc` is assigned to the `_ptr` field. If I just assign 
 it to a variable and don't assign anything to `_ptr`, it will 
 work!

 Is there any possible that there is a compiler bug? I do use 
 ldc2 and `betterC`!


I've had an error message like that before. This was the answer: 
https://stackoverflow.com/questions/46803671/sysmalloc-assertion

Without additional code it's hard to say if that's your problem.
Sep 08 2023
prev sibling next sibling parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 07:59:37 UTC, rempas wrote:

 [I do have ... I do use ldc2 and `betterC`!]


For anyone who is still following, first of all thanks! Second, I 
have bad news and good news!

The bad news is that I tested in an Linux Mint system (mine is an 
Arch Linux) and the it still doesn't work. So unless it's a bug 
in EVERY glibc on Linux, it will take me days to find it at best 
(I can feel the burnout coming!)...

The good news is that I have 
[uploaded](https://codeberg.org/rempas/nemesis) the source code 
and as I have already asked for help for that codebase in the 
past and I'll probably ask in the future, I now have a proper 
codebase to showcase when the "smallest possible example" is not 
enough!

More specifically, look at the showcases 
[Vec](https://codeberg.org/rempas/nemesis/src/branch/dev/source/erebos/s
ructs/vector.d#L44) struct and the
[template](https://codeberg.org/rempas/nemesis/src/branch/dev/source/erebos/
tructs/array.d#L38) that it expands on its body!

The test that fails is for a function called 
[split](https://codeberg.org/rempas/nemesis/src/branch/dev/source/erebos/al
orithms/split.d#L6) and it is located
[here](https://codeberg.org/rempas/nemesis/src/branch/dev/source/tests/algorith
s/test_split.d#L21) (I specifically point to the line that fails inside the
test).

Once again, thank you all for the support!
Sep 08 2023
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Friday, 8 September 2023 at 07:59:37 UTC, rempas wrote:

 I do have the following struct:

 [...]

 Is there any possible that there is a compiler bug? I do use 
 ldc2 and `betterC`!


Could this be a problem of copy construction ?
Sep 08 2023
parent reply rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 16:02:36 UTC, Basile B. wrote:

 Could this be a problem of copy construction ?


I don't think so. The assertion seems to be violated when 
`malloc` is used. And when I assert the result in the `_ptr` 
field. Really weird...
Sep 08 2023
next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Fri, Sep 08, 2023 at 06:59:21PM +0000, rempas via Digitalmars-d-learn wrote:

 On Friday, 8 September 2023 at 16:02:36 UTC, Basile B. wrote:

 
 Could this be a problem of copy construction ?


 
 I don't think so. The assertion seems to be violated when `malloc` is used.
 And when I assert the result in the `_ptr` field. Really weird...


The error message looks to me like a corruption of the malloc heap.
These kinds of bugs are very hard to trace, because they may go
undetected and only show up in specific circumstances, so small
perturbations of completely unrelated code may make the bug appear or
disappear -- just because the bug doesn't show up when you disable some
code does not prove that that's where the problem is; it could be that
corruption is still happening, it just so happens that it goes unnoticed
when the behaviour of the code changes slightly.

My guess is that you have a double-free somewhere, or there's a buffer
overrun. Or maybe some bad interaction with the GC, e.g. if you tried to
free a pointer from the GC heap. (Note that this may not immediately
show up; free() could've assumed that everything was OK when it has in
fact messed up its internal data structures; the problem would only show
up later on in code that's actually unrelated to the real problem.)

If I were in your shoes I'd use Valgrind / Memcheck to try to find the
real cause of the problem.  Chances are, it may have nothing to do with
the bit of code you quoted at all.  You could try to insert extra
malloc/free's in various places around the code (in places along the
code path, but unrelated to the problematic code) to see if that changes
the behaviour of the bug. If it does, your corruption is likely
somewhere other than the _ptr code you showed.


T

-- 
If the comments and the code disagree, it's likely that *both* are wrong. --
Christopher
Sep 08 2023
next sibling parent Basile B. <b2.temp gmx.com> writes:
On Friday, 8 September 2023 at 19:14:47 UTC, H. S. Teoh wrote:

 [...]
 My guess is that you have a double-free somewhere, or there's a 
 buffer overrun. Or maybe some bad interaction with the GC, e.g. 
 if you tried to free a pointer from the GC heap.


That cant be a GC problem as rempas project is compiled with 
`-betterC`
Sep 08 2023
prev sibling next sibling parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 19:14:47 UTC, H. S. Teoh wrote:

 The error message looks to me like a corruption of the malloc 
 heap. These kinds of bugs are very hard to trace, because they 
 may go undetected and only show up in specific circumstances, 
 so small perturbations of completely unrelated code may make 
 the bug appear or disappear -- just because the bug doesn't 
 show up when you disable some code does not prove that that's 
 where the problem is; it could be that corruption is still 
 happening, it just so happens that it goes unnoticed when the 
 behaviour of the code changes slightly.


Yep! That's what I guess as well! Tbh, I knew that playing with 
memory and trying to write a system library will be a touch job 
and I'm up for it! But these bugs really burn me out because like 
you said, nobody can truly help because they are so weird...


 My guess is that you have a double-free somewhere, or there's a 
 buffer overrun. Or maybe some bad interaction with the GC, e.g. 
 if you tried to free a pointer from the GC heap. (Note that 
 this may not immediately show up; free() could've assumed that 
 everything was OK when it has in fact messed up its internal 
 data structures; the problem would only show up later on in 
 code that's actually unrelated to the real problem.)


I have commented out every `free` at this point just to be sure 
and still the problem remains. I don't know what a "buffer 
overrun" is, I will make my research and I will reply you when I 
try. The GC does not exist as I'm `betterC`.


 If I were in your shoes I'd use Valgrind / Memcheck to try to 
 find the real cause of the problem.  Chances are, it may have 
 nothing to do with the bit of code you quoted at all.  You 
 could try to insert extra malloc/free's in various places 
 around the code (in places along the code path, but unrelated 
 to the problematic code) to see if that changes the behaviour 
 of the bug. If it does, your corruption is likely somewhere 
 other than the _ptr code you showed.


 T


Thanks for the advice! I already used Valgrind before I bother 
you guys but because at this point of development, I didn't cared 
about freeing the memory, "valgrind" points so many errors that 
it isn't useful to help me identify what's wrong.
Sep 09 2023
prev sibling parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 19:14:47 UTC, H. S. Teoh wrote:

 My guess is that you have a double-free somewhere, or there's a 
 buffer overrun. Or maybe some bad interaction with the GC, e.g. 
 if you tried to free a pointer from the GC heap. (Note that 
 this may not immediately show up; free() could've assumed that 
 everything was OK when it has in fact messed up its internal 
 data structures; the problem would only show up later on in 
 code that's actually unrelated to the real problem.)


Wait! I did searched on the web! I cannot find anything about a 
"buffer overrun"! I only find about "buffer overflow".
Sep 09 2023
prev sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Friday, 8 September 2023 at 18:59:21 UTC, rempas wrote:

 On Friday, 8 September 2023 at 16:02:36 UTC, Basile B. wrote:

 Could this be a problem of copy construction ?


 I don't think so.


My idea was that if you dont have defined a copy constructor and 
if an instance is assigned to another, then that other instance 
share the same pointer, which can cause memory errors.

To eliminate that risk and to detect where default post-blitting 
may happen you can add

```d
 disable this(this);
```

to the struct.
Sep 08 2023
parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 19:48:33 UTC, Basile B. wrote:

 My idea was that if you dont have defined a copy constructor 
 and if an instance is assigned to another, then that other 
 instance share the same pointer, which can cause memory errors.

 To eliminate that risk and to detect where default 
 post-blitting may happen you can add

 ```d
  disable this(this);
 ```

 to the struct.


Thank you! I did and I got a bunch of errors! Seeing where they 
are located, it doesn't seem to be in the file and instances that 
give me the error. Something really weird happens.

I'll probably return to my own memory allocated and fix the bugs 
I had there, it will probably take me less time...
Sep 09 2023
prev sibling next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 08/09/2023 7:59 PM, rempas wrote:

 |Fatal glibc error: malloc.c:2594 (sysmalloc): assertion failed: 
 (old_top == initial_top (av) && old_size == 0) || ((unsigned long) 
 (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) 
 old_end & (pagesize - 1)) == 0)|


I would strongly suggest that you log all memory sizes that are 
allocated, and double check that you do free.

Also turn on ASAN in ldc.

http://johanengelen.github.io/ldc/2017/12/25/LDC-and-AddressSanitizer.html
Sep 08 2023
parent rempas <rempas tutanota.com> writes:
On Friday, 8 September 2023 at 16:17:15 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 I would strongly suggest that you log all memory sizes that are 
 allocated, and double check that you do free.

 Also turn on ASAN in ldc.

 http://johanengelen.github.io/ldc/2017/12/25/LDC-and-AddressSanitizer.html


Aha! In the search I made about this problem, I've seen a thread 
talking about "sanatizer=address" but only in C. I searched how I 
can use this one on LDC and I wasn't able to find it so, thank 
you!

Unfortunately, I didn't helped because it has the same kind of 
output with [Valgrind](https://valgrind.org/) so it isn't very 
useful in my case...
Sep 08 2023
prev sibling next sibling parent reply Brad Roberts <braddr puremagic.com> writes:
On 9/8/2023 12:59 AM, rempas via Digitalmars-d-learn wrote:

 u64 _cap = 0;   // Total amount of elements (not bytes) we can 



 this._ptr = cast(T*)malloc(size);


I'm pretty sure this is your problem.  You're allocating size bytes 
which is only going to work where sizeof(T) == 1.  Changing to 
malloc(size * sizeof(T)) is likely going to work better.
Sep 09 2023
parent reply rempas <rempas tutanota.com> writes:
On Saturday, 9 September 2023 at 08:54:14 UTC, Brad Roberts wrote:

 I'm pretty sure this is your problem.  You're allocating size 
 bytes which is only going to work where sizeof(T) == 1.  
 Changing to malloc(size * sizeof(T)) is likely going to work 
 better.


Oh man!!!! That was it! I had forget about that! Funny enough, 
the reallocation tests I do letter when expanding the vector do 
include that but I had forgot to place it in the new (because I 
had the an old one and it included this) constructor I had made 
that only allocates memory!

Now, if only one could expect how and why "libc" knows that and 
doesn't just care to give me the memory I asked it for? Or it 
could be than D does something additional without telling us? 
Which can explain when this memory is only present when I assign 
the value to the "this._ptr` field!

Hope I don't have other memory related errors until I finish the 
compiler...

How can I thank you????
Sep 09 2023
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Saturday, 9 September 2023 at 09:21:32 UTC, rempas wrote:


 Now, if only one could expect how and why "libc" knows that and 
 doesn't just care to give me the memory I asked it for? Or it 
 could be than D does something additional without telling us? 
 Which can explain when this memory is only present when I 
 assign the value to the "this._ptr` field!


You are focusing on the wrong problem.

You asked for size bytes, and malloc gave you size bytes. It 
doesn't "know" anything special.

Then you proceeded at some point to write *past* the size bytes. 
What did you overwrite? Probably some internal malloc 
implementation structure. Then it later noticed "hey, this 
structure doesn't make sense, I'm going to report it to the 
user!" That's why you see the message.

Memory problems are very difficult to find, and typically an 
error is triggered far away from the source, in seemingly 
unrelated code. This is why whenever I see an error that smells 
like memory corruption, I stop all other work and find it. Memory 
errors can come and go based on random chance or how the compiler 
lays out functions. So having it "just go away" isn't enough. 
Very very infrequently, this happens because of a codegen issue, 
but most of the time it's pilot error.

-Steve
Sep 09 2023
parent rempas <rempas tutanota.com> writes:
On Saturday, 9 September 2023 at 09:47:14 UTC, Steven 
Schveighoffer wrote:

 You are focusing on the wrong problem.

 You asked for size bytes, and malloc gave you size bytes. It 
 doesn't "know" anything special.

 Then you proceeded at some point to write *past* the size 
 bytes. What did you overwrite? Probably some internal malloc 
 implementation structure. Then it later noticed "hey, this 
 structure doesn't make sense, I'm going to report it to the 
 user!" That's why you see the message.

 Memory problems are very difficult to find, and typically an 
 error is triggered far away from the source, in seemingly 
 unrelated code. This is why whenever I see an error that smells 
 like memory corruption, I stop all other work and find it. 
 Memory errors can come and go based on random chance or how the 
 compiler lays out functions. So having it "just go away" isn't 
 enough. Very very infrequently, this happens because of a 
 codegen issue, but most of the time it's pilot error.

 -Steve


I understand! Thank you for the valuable information. I do have 
lots of things to learn it seems. But that means that I also 
won't get bored anytime soon ;)
Sep 09 2023
prev sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Sat, Sep 09, 2023 at 09:21:32AM +0000, rempas via Digitalmars-d-learn wrote:

 On Saturday, 9 September 2023 at 08:54:14 UTC, Brad Roberts wrote:

 I'm pretty sure this is your problem.  You're allocating size bytes
 which is only going to work where sizeof(T) == 1.  Changing to
 malloc(size * sizeof(T)) is likely going to work better.


 
 Oh man!!!! That was it! I had forget about that! Funny enough, the
 reallocation tests I do letter when expanding the vector do include
 that but I had forgot to place it in the new (because I had the an old
 one and it included this) constructor I had made that only allocates
 memory!
 
 Now, if only one could expect how and why "libc" knows that and
 doesn't just care to give me the memory I asked it for? Or it could be
 than D does something additional without telling us? Which can explain
 when this memory is only present when I assign the value to the
 "this._ptr` field!


libc doesn't know what you intended. All it knows is that you asked it
for 20 bytes (even though you actually needed 40), then later on its
internal structures are corrupted (because you thought you got 40 bytes;
storing data past the 20 bytes overwrote some of malloc's internal data
-- this is the buffer overrun / buffer overflow I referred to). So it
aborts the program instead of continuing to run in a compromised state.


T

-- 
There are four kinds of lies: lies, damn lies, and statistics.
Sep 09 2023
parent rempas <rempas tutanota.com> writes:
On Saturday, 9 September 2023 at 09:56:59 UTC, H. S. Teoh wrote:

 libc doesn't know what you intended. All it knows is that you 
 asked it for 20 bytes (even though you actually needed 40), 
 then later on its internal structures are corrupted (because 
 you thought you got 40 bytes; storing data past the 20 bytes 
 overwrote some of malloc's internal data -- this is the buffer 
 overrun / buffer overflow I referred to). So it aborts the 
 program instead of continuing to run in a compromised state.


 T


Thank you! I fully realize now what's the problem! And that was 
indeed a very sneaky problem. The good news is that I'm mostly 
done with these memory structures and functions so I will 
probably take a while since I find something similar.

I'm lucky there are people smarter, more experience that are 
willing to help. Bless you all and have a great day!
Sep 09 2023
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Friday, 8 September 2023 at 07:59:37 UTC, rempas wrote:

 I do have the following struct:


...

 That's some minimal code that I do have just to showcase it.


This is not ideal. Why? Because 99% of the time, a poster has 
come here with a problem they don't know how to solve, and have 
focused in on where they *think* the problem is. However, the 
problem isn't there. But us reading the description can only see 
what the poster sees, and either don't see a problem ("I'm just 
as confused as you are!") or know there is more to the story.

Not only that, but frequently not-complete code is... not 
complete. And people tend to focus on problems they can see (e.g. 
where is that `_len` defined?), frustrating the poster with 
"trivial" problems that are solved "in the real code".

Inevitably, there is a subsequent post with the real code, and 
that contains the problem.

The best thing to post is a minimally reproducing example. The 
next best thing is a link to a complex reproducing example. Which 
you have done later (I will take a look). I just wanted to point 
this out because it's a frequent problem on these forums.


 So, some times, this work will works, some others, it will give 
 me the following error:

 `Fatal glibc error: malloc.c:2594 (sysmalloc): assertion 
 failed: (old_top == initial_top (av) && old_size == 0) || 
 ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) 
 && ((unsigned long) old_end & (pagesize - 1)) == 0)`


This is an internal message from glibc. It seems the malloc 
structure is corrupted.


 Is there any possible that there is a compiler bug? I do use 
 ldc2 and `betterC`!


There is always a chance...

Now, critiquing your original code, I see red flags here:


 ```d
   u64 _cap = 0;   // Total amount of elements (not bytes) we 
 can store
 ```


and then later:


 ```d
   this(i64 size) {
     this._len = 0;
     this._cap = size;
 ```


ok, so `size` must mean the number of elements, not the number of 
bytes.


 ```d
     static if (is(T == char)) { size += 1; } // Additional 
 space for the null terminator
     this._ptr = cast(T*)malloc(size);
 ```


Here, you have allocated `size` bytes for the array. Is this what 
is intended? Your comments suggest otherwise! If `T.sizeof` is 2 
or more, then you still only allocate e.g. 20 bytes for a `_cap` 
of 20. but an array of 20 T's would require 40 bytes.

-Steve
Sep 09 2023
parent reply rempas <rempas tutanota.com> writes:
On Saturday, 9 September 2023 at 09:04:18 UTC, Steven 
Schveighoffer wrote:

 This is not ideal. Why? Because 99% of the time, a poster has 
 come here with a problem they don't know how to solve, and have 
 focused in on where they *think* the problem is. However, the 
 problem isn't there. But us reading the description can only 
 see what the poster sees, and either don't see a problem ("I'm 
 just as confused as you are!") or know there is more to the 
 story.

 Not only that, but frequently not-complete code is... not 
 complete. And people tend to focus on problems they can see 
 (e.g. where is that `_len` defined?), frustrating the poster 
 with "trivial" problems that are solved "in the real code".

 Inevitably, there is a subsequent post with the real code, and 
 that contains the problem.

 The best thing to post is a minimally reproducing example. The 
 next best thing is a link to a complex reproducing example. 
 Which you have done later (I will take a look). I just wanted 
 to point this out because it's a frequent problem on these 
 forums.


Yeah... I got my lesson! Tbh, I wanted to avoid posting my source 
as much as possible because I do have this "perfectionist" 
mindset (which I am aware is bad and trying to get rid of, and 
have already done progress) and I want the source to sound as 
"professional" as possible as I take serious my project and have 
hopes for it! But yeah, NEVER again!


 Here, you have allocated `size` bytes for the array. Is this 
 what is intended? Your comments suggest otherwise! If 
 `T.sizeof` is 2 or more, then you still only allocate e.g. 20 
 bytes for a `_cap` of 20. but an array of 20 T's would require 
 40 bytes.

 -Steve


Bingo! You and Brad found out! And like I said to him, it's funny 
that I had a previous function that just allocates memory and I 
removed it and then created that one and in that one, I forgot 
about that. It's super funny cause I even wrote the comment that 
explains it and my reallocation function (that gets triggered 
when someone wants to expand the collection) has that 
multiplication one it.

Thank you and everyone else for the help! Hope I can at least 
finish and share something that people will enjoy using!
Sep 09 2023
parent reply bachmeier <no spam.net> writes:
On Saturday, 9 September 2023 at 09:30:10 UTC, rempas wrote:


 Bingo! You and Brad found out!


Hate to be that guy, but I posted a link to a stackoverflow 
question with the exact error message you were getting, and the 
solution. And I told you I had experienced the same error and 
that question fixed it.
Sep 09 2023
parent rempas <rempas tutanota.com> writes:
On Saturday, 9 September 2023 at 10:54:34 UTC, bachmeier wrote:

 Hate to be that guy, but I posted a link to a stackoverflow 
 question with the exact error message you were getting, and the 
 solution. And I told you I had experienced the same error and 
 that question fixed it.


No reason to not says something, always speak you mind my friend!

I did read it and all the replies but the problem is that, I was 
so focused on thinking that something is going wrong with the 
compiler or with libc or something unexpected. For that reason, I 
made the mistake to not see other things that might had gotten 
wrong even tho you literally gave me the answer! That's another 
problem of mine that I am aware and trying to fix (not only when 
it comes to coding but in general) and I do apologize for wasting 
both mine and everyone else's time!

Now that people explained to me, I fully understood what's going 
on and how I made the mistake. Have a great day my friend and I 
hope I can repay one day! ;)
Sep 09 2023