www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Please fix `.init` property

reply Hipreme <msnmancini hotmail.com> writes:
I was doing a completely functional code, then I noticed some 
variable changing abruptly to a strange value. After the first 
run in a function, the value completely changed, I've reduced and 
found that I was able to actually modify `.init`.

```d
struct MyTest
{
     string[] dirs = ["source"];
}

void DoIt(ref MyTest a )
{
     a.dirs[0] = null;
}

void main()
{
     MyTest t;
     DoIt(t);
     MyTest t2;
     assert(t2.dirs == MyTest.init.dirs); //Crashes
}
```

Looks like everyone knew this bug existed, but it caused me waste 
4 hours trying to find when did it happen. I can't understand why 
this has not been fixed yet, but this leads to extremely erratic 
behaviour, which I'm going to do an ugly workaround.

Since we are on a year of stability, I think getting this kind of 
bug fixed could be a priority.
Jan 07
next sibling parent reply matheus <matheus gmail.com> writes:
On Sunday, 7 January 2024 at 23:12:30 UTC, Hipreme wrote:

 ...


For what it seems this happens only with arrays?

This happened with for example int[] a; but when I tried with 
other types (Non-arrays) like (string, int) this problem didn't 
occur.

Since like in this example string arrays are pointers, I think 
something was messed-up with the address.

Matheus.
Jan 07
next sibling parent matheus <matheus gmail.com> writes:
On Monday, 8 January 2024 at 04:14:48 UTC, matheus wrote:

 ...
 Since like in this example string arrays are pointers, I think 
 something was messed-up with the address.


Correction: Something is messing-up with the address.

Matheus.
Jan 07
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, January 7, 2024 9:14:48 PM MST matheus via Digitalmars-d wrote:

 On Sunday, 7 January 2024 at 23:12:30 UTC, Hipreme wrote:

 ...


 For what it seems this happens only with arrays?

 This happened with for example int[] a; but when I tried with
 other types (Non-arrays) like (string, int) this problem didn't
 occur.

 Since like in this example string arrays are pointers, I think
 something was messed-up with the address.

 Matheus.


Well, to an extent, the problem here is simply that the member variable is a
type with mutable indirections. The dynamic array itself is just a pointer
and a length, and mutating what it points to like the OP's example doesn't
actually mutate the init value. In order to avoid that sort of mutation,
initializing the struct would need to give you a deep copy rather than a
shallow copy. Other member variables with mutable indirections would have
similar problems. For instance, all of the assertions in this code pass:

class C
{
    string foo = "foo";
}

struct S
{
    C c = new C;
}

void main()
{
    S s1;
    assert(s1.c.foo == "foo");
    assert(s1.c.foo == S.init.c.foo);

    S s2;
    s2.c.foo = "bar";
    assert(s2.c.foo == "bar");

    assert(s1.c.foo == "bar");
    assert(S.init.c.foo == "bar");

    assert((cast(void*)s1.c) is cast(void*)s2.c);
    assert((cast(void*)s1.c) is cast(void*)S.init.c);

    S s3;
    s3.c = new C;
    assert(s3.c.foo == "foo");

    assert(s1.c.foo == "bar");
    assert(s2.c.foo == "bar");
    assert(S.init.c.foo == "bar");

    assert((cast(void*)s1.c) is cast(void*)s2.c);
    assert((cast(void*)s1.c) is cast(void*)S.init.c);

    assert((cast(void*)s3.c) !is cast(void*)S.init.c);
}

The init value itself is never mutated, but what it points to is. Someone
could interpret it like the init value had mutated, because what it's
pointed to changed, and therefore, accessing the values via init then
results in different values that were there at the start of the program, but
technically, init itself never changed.

As such, the OP should expect that mutating values in the array would affect
any other variable when it's default-initialized, and I don't think that
it's realistic for it to work any other way, because that requires making a
deep copy of the init value rather than a shallow copy, and that's not
something that can be done in the general case, because D doesn't have a
mechanism for that.

However, in spite of all of that, something weird is going on with the OP's
example, because while variables that are created after the element of the
array has been mutated see that mutation, the init value itself does not.
So, it would appear that the array in the init value itself somehow ends up
pointing to a different block of memory than the array does when the struct
is default initilaized. This example shows the same problem

struct S
{
    int[] arr = [1, 2, 3];
}

void main()
{
    S s1;
    assert(s1.arr == [1, 2, 3]);
    assert(s1.arr == S.init.arr);

    S s2;
    s2.arr[0] = 42;
    assert(s2.arr == [42, 2, 3]);

    assert(s1.arr == [42, 2, 3]);
    assert(S.init.arr == [1, 2, 3]);
}

and if I were to change main to

void main()
{
    import std.stdio;
    writeln(S.init.arr.ptr);

    S s1;
    writeln(s1.arr.ptr);

    S s2;
    writeln(s2.arr.ptr);

    writeln(S.init.arr.ptr);
}

running it on my computer results in

C378D421000
2C3F08
2C3F08
C378D421010

So, for some reason, the ptr of the array in the init value has a different
address from the one in the struct's after they've been default-initialized,
but the struct's get the same pointer value.

This is in stark contrast to my previous example where the member variable
that was a class reference ended up with the address being the same for both
the init value and the default-initialized structs. So, it would appear that
the compiler or runtime is doing something different with dynamic arrays
than it is with classes, and the behavior definitely seems wrong, because it
means that a default-initialized struct does not match its init value - and
this without worrying about mutating anything. e.g. this assertion fails
when it should never fail

struct S
{
    int[] arr = [1, 2, 3];
}

void main()
{
    S s;
    assert(s is S.init);
}

So, I would say that there is definitely a bug here with regards to init
values when a struct has a member variable which is a dynamic array.

However, ultimately, that's not really what the OP seems to be complaining
about. Rather, he seems to be complaining about the fact that if you mutate
a member variable with mutable indirections which were not null in the init
value, then the changes to those mutable indirections are visible via the
init value, and I don't think that that's something that's ever going to
change, because it would require that default initialization do a deep copy
rather than a shallow copy. Obviously, it can be surprising if it's not
something that you've run into or thought through before, but D doesn't have
a generic way to do deep copies, and you wouldn't necessarily want a deep
copy in all cases anyway, so even if we could make it do a deep copy, that
wouldn't necessarily be a good change for the language as a whole, though it
would certainly fix the issue that the OP ran into.

Arguably, what the OP needs is a default constructor rather than an init
value, but that's not the way that D is designed, and it would be
problematic with some of its features to use default construction instead
(though this is far from the only case where not having default construction
for structs can be annoying).

- Jonathan M Davis
Jan 07
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 8 January 2024 at 05:22:20 UTC, Jonathan M Davis wrote:

 On Sunday, January 7, 2024 9:14:48 PM MST matheus via 
 Digitalmars-d wrote:

 On Sunday, 7 January 2024 at 23:12:30 UTC, Hipreme wrote:

 ...


 For what it seems this happens only with arrays?

 This happened with for example int[] a; but when I tried with
 other types (Non-arrays) like (string, int) this problem didn't
 occur.

 Since like in this example string arrays are pointers, I think 
 something was messed-up with the address.

 Matheus.


 Well, to an extent, the problem here is simply that the member 
 variable is a type with mutable indirections. The dynamic array 
 itself is just a pointer and a length, and mutating what it 
 points to like the OP's example doesn't actually mutate the 
 init value. In order to avoid that sort of mutation, 
 initializing the struct would need to give you a deep copy 
 rather than a shallow copy. Other member variables with mutable 
 indirections would have similar problems. For instance, all of 
 the assertions in this code pass:

 class C
 {
     string foo = "foo";
 }

 struct S
 {
     C c = new C;
 }

 void main()
 {
     S s1;
     assert(s1.c.foo == "foo");
     assert(s1.c.foo == S.init.c.foo);

     S s2;
     s2.c.foo = "bar";
     assert(s2.c.foo == "bar");

     assert(s1.c.foo == "bar");
     assert(S.init.c.foo == "bar");

     assert((cast(void*)s1.c) is cast(void*)s2.c);
     assert((cast(void*)s1.c) is cast(void*)S.init.c);

     S s3;
     s3.c = new C;
     assert(s3.c.foo == "foo");

     assert(s1.c.foo == "bar");
     assert(s2.c.foo == "bar");
     assert(S.init.c.foo == "bar");

     assert((cast(void*)s1.c) is cast(void*)s2.c);
     assert((cast(void*)s1.c) is cast(void*)S.init.c);

     assert((cast(void*)s3.c) !is cast(void*)S.init.c);
 }

 The init value itself is never mutated, but what it points to 
 is. Someone could interpret it like the init value had mutated, 
 because what it's pointed to changed, and therefore, accessing 
 the values via init then results in different values that were 
 there at the start of the program, but technically, init itself 
 never changed.

 As such, the OP should expect that mutating values in the array 
 would affect any other variable when it's default-initialized, 
 and I don't think that it's realistic for it to work any other 
 way, because that requires making a deep copy of the init value 
 rather than a shallow copy, and that's not something that can 
 be done in the general case, because D doesn't have a mechanism 
 for that.

 However, in spite of all of that, something weird is going on 
 with the OP's example, because while variables that are created 
 after the element of the array has been mutated see that 
 mutation, the init value itself does not. So, it would appear 
 that the array in the init value itself somehow ends up 
 pointing to a different block of memory than the array does 
 when the struct is default initilaized. This example shows the 
 same problem

 struct S
 {
     int[] arr = [1, 2, 3];
 }

 void main()
 {
     S s1;
     assert(s1.arr == [1, 2, 3]);
     assert(s1.arr == S.init.arr);

     S s2;
     s2.arr[0] = 42;
     assert(s2.arr == [42, 2, 3]);

     assert(s1.arr == [42, 2, 3]);
     assert(S.init.arr == [1, 2, 3]);
 }

 and if I were to change main to

 void main()
 {
     import std.stdio;
     writeln(S.init.arr.ptr);

     S s1;
     writeln(s1.arr.ptr);

     S s2;
     writeln(s2.arr.ptr);

     writeln(S.init.arr.ptr);
 }

 running it on my computer results in

 C378D421000
 2C3F08
 2C3F08
 C378D421010

 So, for some reason, the ptr of the array in the init value has 
 a different address from the one in the struct's after they've 
 been default-initialized, but the struct's get the same pointer 
 value.

 This is in stark contrast to my previous example where the 
 member variable that was a class reference ended up with the 
 address being the same for both the init value and the 
 default-initialized structs. So, it would appear that the 
 compiler or runtime is doing something different with dynamic 
 arrays than it is with classes, and the behavior definitely 
 seems wrong, because it means that a default-initialized struct 
 does not match its init value - and this without worrying about 
 mutating anything. e.g. this assertion fails when it should 
 never fail

 struct S
 {
     int[] arr = [1, 2, 3];
 }

 void main()
 {
     S s;
     assert(s is S.init);
 }

 So, I would say that there is definitely a bug here with 
 regards to init values when a struct has a member variable 
 which is a dynamic array.

 However, ultimately, that's not really what the OP seems to be 
 complaining about. Rather, he seems to be complaining about the 
 fact that if you mutate a member variable with mutable 
 indirections which were not null in the init value, then the 
 changes to those mutable indirections are visible via the init 
 value, and I don't think that that's something that's ever 
 going to change, because it would require that default 
 initialization do a deep copy rather than a shallow copy. 
 Obviously, it can be surprising if it's not something that 
 you've run into or thought through before, but D doesn't have a 
 generic way to do deep copies, and you wouldn't necessarily 
 want a deep copy in all cases anyway, so even if we could make 
 it do a deep copy, that wouldn't necessarily be a good change 
 for the language as a whole, though it would certainly fix the 
 issue that the OP ran into.

 Arguably, what the OP needs is a default constructor rather 
 than an init value, but that's not the way that D is designed, 
 and it would be problematic with some of its features to use 
 default construction instead (though this is far from the only 
 case where not having default construction for structs can be 
 annoying).

 - Jonathan M Davis


Force default values to be typed immutable?
Jan 08
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, January 8, 2024 3:57:52 AM MST FeepingCreature via Digitalmars-d 
wrote:

 However, ultimately, that's not really what the OP seems to be
 complaining about. Rather, he seems to be complaining about the
 fact that if you mutate a member variable with mutable
 indirections which were not null in the init value, then the
 changes to those mutable indirections are visible via the init
 value, and I don't think that that's something that's ever
 going to change, because it would require that default
 initialization do a deep copy rather than a shallow copy.
 Obviously, it can be surprising if it's not something that
 you've run into or thought through before, but D doesn't have a
 generic way to do deep copies, and you wouldn't necessarily
 want a deep copy in all cases anyway, so even if we could make
 it do a deep copy, that wouldn't necessarily be a good change
 for the language as a whole, though it would certainly fix the
 issue that the OP ran into.

 Arguably, what the OP needs is a default constructor rather
 than an init value, but that's not the way that D is designed,
 and it would be problematic with some of its features to use
 default construction instead (though this is far from the only
 case where not having default construction for structs can be
 annoying).


 Force default values to be typed immutable?


That would require that there be a general way to copy an immutable value to
a mutable one, because the object must be mutable after it's been
default-initialized.

Also, if any sort of deep-copying occurs, then you end up in the bizarre
situation where code like

S s;
assert(s is S.init);

fails, which is also problematic.

Bastiaan's suggestion to make such cases illegal is probably the only sane
way to prevent these kind of issues, though I think that it would have to be
an error to have mutable indirections with an allocation rather than illegal
to point to any allocations; otherwise, stuff like SysTime's member which is
Rebindable(immutable TimeZone) becomes illegal.

- Jonathan M Davis
Jan 08
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 8 January 2024 at 18:03:00 UTC, Jonathan M Davis wrote:

 On Monday, January 8, 2024 3:57:52 AM MST FeepingCreature via 
 Digitalmars-d wrote:

 Force default values to be typed immutable?


 That would require that there be a general way to copy an 
 immutable value to a mutable one, because the object must be 
 mutable after it's been default-initialized.

 Bastiaan's suggestion to make such cases illegal is probably 
 the only sane way to prevent these kind of issues, though I 
 think that it would have to be an error to have mutable 
 indirections with an allocation rather than illegal to point to 
 any allocations; otherwise, stuff like SysTime's member which 
 is Rebindable(immutable TimeZone) becomes illegal.

 - Jonathan M Davis


Yes, the point of making it forced `immutable` is exactly to make 
the problematic cases illegal. Don't even think about deepcopy vs 
array vs object etc., just let the const system handle it.
Jan 08
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, January 8, 2024 11:16:42 PM MST FeepingCreature via Digitalmars-d 
wrote:

 On Monday, 8 January 2024 at 18:03:00 UTC, Jonathan M Davis wrote:

 On Monday, January 8, 2024 3:57:52 AM MST FeepingCreature via

 Digitalmars-d wrote:

 Force default values to be typed immutable?


 That would require that there be a general way to copy an
 immutable value to a mutable one, because the object must be
 mutable after it's been default-initialized.

 Bastiaan's suggestion to make such cases illegal is probably
 the only sane way to prevent these kind of issues, though I
 think that it would have to be an error to have mutable
 indirections with an allocation rather than illegal to point to
 any allocations; otherwise, stuff like SysTime's member which
 is Rebindable(immutable TimeZone) becomes illegal.


 Yes, the point of making it forced `immutable` is exactly to make
 the problematic cases illegal. Don't even think about deepcopy vs
 array vs object etc., just let the const system handle it.


Maybe? My gut reaction is that there's likely to be an issue with that, but
at a glance, it seems like it might work. So, it could be a really good
idea, but there may also be some subtlety that I'm not thinking of at the
moment.

- Jonathan M Davis
Jan 09
prev sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Tuesday, 9 January 2024 at 06:16:42 UTC, FeepingCreature wrote:

 Yes, the point of making it forced `immutable` is exactly to 
 make the problematic cases illegal. Don't even think about 
 deepcopy vs array vs object etc., just let the const system 
 handle it.


I might be misinterpreting, but it seems like you're suggesting
```
S s;
s.arr[0] = "imm"; // compile-time error?
s.arr = ["new"];
s.arr[0] = "mut"; // ok
```
I'm not seeing how that would fly.

I can imagine it being easier done at runtime - with a bit of 
extra overhead - using mprotect or tagged pointers.
Jan 12
next sibling parent Hipreme <msnmancini hotmail.com> writes:
On Friday, 12 January 2024 at 09:57:25 UTC, Iain Buclaw wrote:

 On Tuesday, 9 January 2024 at 06:16:42 UTC, FeepingCreature 
 wrote:

 Yes, the point of making it forced `immutable` is exactly to 
 make the problematic cases illegal. Don't even think about 
 deepcopy vs array vs object etc., just let the const system 
 handle it.


 I might be misinterpreting, but it seems like you're suggesting
 ```
 S s;
 s.arr[0] = "imm"; // compile-time error?
 s.arr = ["new"];
 s.arr[0] = "mut"; // ok
 ```
 I'm not seeing how that would fly.

 I can imagine it being easier done at runtime - with a bit of 
 extra overhead - using mprotect or tagged pointers.



I see no problem in having runtime overhead for the sake of 
having default initialized arrays in that case :)

I also would not complain if we had a way to choose between the 
extra overhead vs an error message by using an UDA.
Jan 12
prev sibling parent FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 12 January 2024 at 09:57:25 UTC, Iain Buclaw wrote:

 On Tuesday, 9 January 2024 at 06:16:42 UTC, FeepingCreature 
 wrote:

 Yes, the point of making it forced `immutable` is exactly to 
 make the problematic cases illegal. Don't even think about 
 deepcopy vs array vs object etc., just let the const system 
 handle it.


 I might be misinterpreting, but it seems like you're suggesting
 ```
 S s;
 s.arr[0] = "imm"; // compile-time error?
 s.arr = ["new"];
 s.arr[0] = "mut"; // ok
 ```
 I'm not seeing how that would fly.

 I can imagine it being easier done at runtime - with a bit of 
 extra overhead - using mprotect or tagged pointers.


Not quite: you couldn't even *define* `S` as

```
struct S
{
     int[] arr = [1, 2, 3]; // Error: immutable(int)[] cannot be 
converted to int[].
}
```

The type of an array-literal, *evaluated at compile-time,* would 
always be immutable. That way, you'd either *have* to define a 
constructor or you would be protected from accidental aliasing 
anyways. You'd be missing out on *intentional* aliasing, but 
that's such a corner case I'm fine sacrificing it.
Jan 13
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, January 7, 2024 10:22:20 PM MST Jonathan M Davis via Digitalmars-d 
wrote:

 So, I would say that there is definitely a bug here with regards to init
 values when a struct has a member variable which is a dynamic array.


https://issues.dlang.org/show_bug.cgi?id=24324

- Jonathan M Davis
Jan 07
prev sibling next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Sunday, 7 January 2024 at 23:12:30 UTC, Hipreme wrote:

 ```d
 struct MyTest
 {
     string[] dirs = ["source"];
 }
 ```


I have argued that initializers of non-static members that 
allocate should be an error [1]. Mike and Steven agree [2, 3].

[1] 
https://forum.dlang.org/post/ogvubzgprghefclgluce forum.dlang.org
[2] 
https://forum.dlang.org/post/wvrasioewzbqrqsufwxd forum.dlang.org
[3] https://forum.dlang.org/post/t7vm2o$p4q$1 digitalmars.com

I will add pointers to 
https://issues.dlang.org/show_bug.cgi?id=24324.

-- Bastiaan.
Jan 08
parent Hipreme <msnmancini hotmail.com> writes:
On Monday, 8 January 2024 at 11:53:52 UTC, Bastiaan Veelo wrote:

 On Sunday, 7 January 2024 at 23:12:30 UTC, Hipreme wrote:

 ```d
 struct MyTest
 {
     string[] dirs = ["source"];
 }
 ```


 I have argued that initializers of non-static members that 
 allocate should be an error [1]. Mike and Steven agree [2, 3].

 [1] 
 https://forum.dlang.org/post/ogvubzgprghefclgluce forum.dlang.org
 [2] 
 https://forum.dlang.org/post/wvrasioewzbqrqsufwxd forum.dlang.org
 [3] https://forum.dlang.org/post/t7vm2o$p4q$1 digitalmars.com

 I will add pointers to 
 https://issues.dlang.org/show_bug.cgi?id=24324.

 -- Bastiaan.



I have 2 thoughts on how it could be solved:

- Make it an error
- Make a copy of a shared instance

Currently, the solution for that is doing a copy of a shared 
instance anyway. If it impacts the performance, that is on the 
user. Having this erratic behavior is the only thing that can't 
continue. I've lost a plenty of time in a big project.

And no, almost no one has the entire spec in mind. Even more to 
getting those particular behaviors. The spec can be precise, but 
it doesn't mean it is correct to work like that.
People will continue falling on the same stone, early or late.
Jan 08
prev sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Sun, Jan 07, 2024 at 11:12:30PM +0000, Hipreme via Digitalmars-d wrote:

 I was doing a completely functional code, then I noticed some variable
 changing abruptly to a strange value. After the first run in a
 function, the value completely changed, I've reduced and found that I
 was able to actually modify `.init`.
 
 ```d
 struct MyTest
 {
     string[] dirs = ["source"];
 }


Using initializer syntax to initialize mutable array members is
generally not recommended, because it does not do what you expect. It
does NOT allocate a new array per instance of MyTest; instead, it stores
the array once in the program's preinit data segment, and sets the
default value of .dirs to point to that instance.  This is generally
harmless if the array is immutable, but if the data is mutable, you're
probably in for a surprise.

If you want every instance of the struct to have a fresh copy of the
array, use a ctor instead.

(Whether the current behaviour should be changed is up for debate,
though. This definitely isn't the first time users have run into this.
In fact just today somebody else asked the same question on D.learn. So
it's definitely in the territory of "does not do the expected thing",
which is an indication that the default behaviour was poorly chosen.)


T

-- 
If blunt statements had a point, they wouldn't be blunt...
Jan 08