www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - fieldPostBlit - what is wrong with this and workarounds

reply "Daniel Davidson" <nospam spam.com> writes:
Given this code:

   import plus.tvm.rate_curve;
   struct T {
     RateCurve m;
   }
   struct S {
     const(T) rc;
   }

I get this error: Error: mutable method 
plus.models.dossier.__unittestL42_1.T.__fieldPostBlit is not 
callable using a const object

Is this fundamentally incorrect? I abandoned immutable in hopes 
that const would be more manageable and so far it has. But these 
types of issues still crop up and stop me in my tracks.

What are the requirements on RateCurve and T for this to work?

Thanks
Dan
Oct 31 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Daniel Davidson:


 I get this error: Error: mutable method 
 plus.models.dossier.__unittestL42_1.T.__fieldPostBlit is not 
 callable using a const object


Related:
http://d.puremagic.com/issues/show_bug.cgi?id=4867

Bye,
bearophile
Oct 31 2013
parent "Daniel Davidson" <nospam spam.com> writes:
On Thursday, 31 October 2013 at 14:28:31 UTC, bearophile wrote:

 Daniel Davidson:


 I get this error: Error: mutable method 
 plus.models.dossier.__unittestL42_1.T.__fieldPostBlit is not 
 callable using a const object


 Related:
 http://d.puremagic.com/issues/show_bug.cgi?id=4867

 Bye,
 bearophile


Thanks but I don't see the relation.
Oct 31 2013
prev sibling next sibling parent reply "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 31 October 2013 at 14:03:28 UTC, Daniel Davidson 
wrote:

 Given this code:

   import plus.tvm.rate_curve;
   struct T {
     RateCurve m;
   }
   struct S {
     const(T) rc;
   }

 I get this error: Error: mutable method 
 plus.models.dossier.__unittestL42_1.T.__fieldPostBlit is not 
 callable using a const object

 Is this fundamentally incorrect? I abandoned immutable in hopes 
 that const would be more manageable and so far it has. But 
 these types of issues still crop up and stop me in my tracks.

 What are the requirements on RateCurve and T for this to work?

 Thanks
 Dan


D language is fundamentally incorrect here. There are sufficient 
issues in bugzilla (one which bearophile provided is actually 
relevant, it shows same problem from a little bit different 
aspect) and the problem tends to pop up quire regularly. It looks 
like nothing is done because it is design issue and although 
there were attempts to come up with, no idea is viewed as final 
solution. The easiest way to solve problem is to strip const.
Oct 31 2013
parent reply "Daniel Davidson" <nospam spam.com> writes:
On Thursday, 31 October 2013 at 15:56:45 UTC, Maxim Fomin wrote:

 On Thursday, 31 October 2013 at 14:03:28 UTC, Daniel Davidson 
 wrote:

 Given this code:

  import plus.tvm.rate_curve;
  struct T {
    RateCurve m;
  }
  struct S {
    const(T) rc;
  }

 I get this error: Error: mutable method 
 plus.models.dossier.__unittestL42_1.T.__fieldPostBlit is not 
 callable using a const object

 Is this fundamentally incorrect? I abandoned immutable in 
 hopes that const would be more manageable and so far it has. 
 But these types of issues still crop up and stop me in my 
 tracks.

 What are the requirements on RateCurve and T for this to work?

 Thanks
 Dan


 D language is fundamentally incorrect here. There are 
 sufficient issues in bugzilla (one which bearophile provided is 
 actually relevant, it shows same problem from a little bit 
 different aspect) and the problem tends to pop up quire 
 regularly. It looks like nothing is done because it is design 
 issue and although there were attempts to come up with, no idea 
 is viewed as final solution. The easiest way to solve problem 
 is to strip const.


I'm think you may be correct... but what specifically is 
incorrect about the language?
When I see __fieldPostBlit I get the feeling it is more an 
implementation issue than the language.

If as you suggest, D language is fundamentally broken - is it 
possible we can just get a plan for fixing it? To me this seems 
like a huge design hole and I come back to the threads with 
questions again and again and I really don't know how others 
manage the use of const/immutable.

I've already given up on immutable. If I have to strip const as 
well then that is a huge pain and unfortunate since the concept 
seems sound and well argued in TDPL. Trying to mess with 
mutable/const/immutable on mostly working code is like playing 
whack a mole.
Oct 31 2013
parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Thursday, 31 October 2013 at 17:22:07 UTC, Daniel Davidson 
wrote:

 I'm think you may be correct... but what specifically is 
 incorrect about the language?
 When I see __fieldPostBlit I get the feeling it is more an 
 implementation issue than the language.


__postblit also looks like implementation detail but it is same 
problem. Field postblit means that aggregate has some members 
which have postblits so they should be called. This is same 
problem which blows in higher level.


 If as you suggest, D language is fundamentally broken - is it 
 possible we can just get a plan for fixing it? To me this seems 
 like a huge design hole and I come back to the threads with 
 questions again and again and I really don't know how others 
 manage the use of const/immutable.


This is probable tenth thread about const postblits which I do 
remember for all my (only) 2 years in D. D2 (version currently in 
use) dates back to 2004 AFAIK and Andrei (helping to design the 
language) is in D since 2006, also AFAIK. Draw your own 
conclusions.


 I've already given up on immutable. If I have to strip const as 
 well then that is a huge pain and unfortunate since the concept 
 seems sound and well argued in TDPL. Trying to mess with 
 mutable/const/immutable on mostly working code is like playing 
 whack a mole.


One can do: 1) revise the design to challange whether consts are 
really necessary 2a) use workarounds like const pointer 2b) use 
cast to break const or another tricks (explicit casts are better 
because holes in type system may be fixed in some point in the 
feature)
Oct 31 2013
prev sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, October 31, 2013 15:03:26 Daniel Davidson wrote:

 Given this code:
 
 import plus.tvm.rate_curve;
 struct T {
 RateCurve m;
 }
 struct S {
 const(T) rc;
 }
 
 I get this error: Error: mutable method
 plus.models.dossier.__unittestL42_1.T.__fieldPostBlit is not
 callable using a const object
 
 Is this fundamentally incorrect? I abandoned immutable in hopes
 that const would be more manageable and so far it has. But these
 types of issues still crop up and stop me in my tracks.
 
 What are the requirements on RateCurve and T for this to work?


const and postblit fundamentally don't mix, because for it to work, you have 
to violate the type system. With postblits, the struct gets memcpied and then 
the postblit constructor has the chance to mutate the resulting object to make 
the pieces different that need to be different. However, to do that mutation 
when the object is const would mean mutating a const object which violates the 
type system. What we really need is copy constructors, but they haven't been 
added yet. At one point, Andrei was saying that he and Walter had a solution, 
but he didn't elaborate on it. I assume that it involved introducing copy 
constructors, but I don't know, and this issue has not yet been resolved.

Now, in your particular code example, you don't define postblit constructor, 
but my guess would be that RateCurve defines one, making it so that a postblit 
constructor is generated for T which uses the one for RateCurve, and the S 
gets one which uses T's. And because const doesn't work with postblit 
constructors, S becomes uncopyable, and if there's any code that requires that 
a copy be made, then it'll fail to compile (I'm not sure whether the fact that 
it's uncopyable will result in an error if no attempts to copy it are made, 
but you'll definitely get an error if an attempt to copy is made).

Hopefully, this problem will be resolved, but regardless of that, I would 
advise against ever having a const member variable in a struct. Even if you 
don't have any postblit issues, such a struct can never be assigned to, and it 
becomes essentially unusuable in any situation where you would have to assign 
a value to it (e.g. if it were in an array). You can certainly make a struct's 
member variable const if you want to, but you're going to run into stuff that 
won't work as a result. It's far better to just provide a property for it 
which is const (probably returning the member by const ref if you want to 
avoid copying it when using the property) rather than making the member 
variable itself const.

- Jonathan M Davis
Oct 31 2013
parent reply "Daniel Davidson" <nospam spam.com> writes:
On Thursday, 31 October 2013 at 19:39:44 UTC, Jonathan M Davis 
wrote:

 const and postblit fundamentally don't mix, because for it to 
 work, you have
 to violate the type system. With postblits, the struct gets 
 memcpied and then
 the postblit constructor has the chance to mutate the resulting 
 object to make
 the pieces different that need to be different. However, to do 
 that mutation
 when the object is const would mean mutating a const object 
 which violates the
 type system. What we really need is copy constructors, but they 
 haven't been
 added yet. At one point, Andrei was saying that he and Walter 
 had a solution,
 but he didn't elaborate on it. I assume that it involved 
 introducing copy
 constructors, but I don't know, and this issue has not yet been 
 resolved.


This makes sense, but I think we could have a simpler solution 
that does not involve a copy constructor addition. It seems the 
problem is postblit offers a hole in the system whereby 
developers could change immutable data. Because of that postblit 
and const don't mix. The simple fix would be take the 
implementation of postblit away from the developer. It seems the 
use case for postblit is really only to provide deep copy 
semantics by having developer throw in a bunch of dups where 
desired.
What if the language just provided standard postblit 
functionality not defined by the user. However it is specified, 
it should be easy to implement.

struct S {
   // blit and no postblit dup -> sharing
   int[] y;

   // blit with request for postblit dup -> no sharing
    postblit int[] x;
}

Ideally, [].dup would be made pure. Once that is done, generic 
field by field deep copy could be done and be made pure, thus 
returning optionally immutable data. While it is true the 
generated postblit will be mutating the incipient instance - that 
is fine as it is called in a construction context and dup is 
surely idempotent. Why make the developer write a function for 
simple copy?


 Now, in your particular code example, you don't define postblit 
 constructor,
 but my guess would be that RateCurve defines one, making it so 
 that a postblit
 constructor is generated for T which uses the one for 
 RateCurve, and the S
 gets one which uses T's. And because const doesn't work with 
 postblit
 constructors, S becomes uncopyable, and if there's any code 
 that requires that
 a copy be made, then it'll fail to compile (I'm not sure 
 whether the fact that
 it's uncopyable will result in an error if no attempts to copy 
 it are made,
 but you'll definitely get an error if an attempt to copy is 
 made).


Your guesses are correct.


 Hopefully, this problem will be resolved, but regardless of 
 that, I would
 advise against ever having a const member variable in a struct. 
 Even if you
 don't have any postblit issues, such a struct can never be 
 assigned to, and it
 becomes essentially unusuable in any situation where you would 
 have to assign
 a value to it (e.g. if it were in an array). You can certainly 
 make a struct's
 member variable const if you want to, but you're going to run 
 into stuff that
 won't work as a result. It's far better to just provide a 
 property for it
 which is const (probably returning the member by const ref if 
 you want to
 avoid copying it when using the property) rather than making 
 the member
 variable itself const.

 - Jonathan M Davis


I am now convinced avoiding `const(T) t` as a member is wise 
advice. The suggestion of leaving it non-const and providing a 
const accessor is good - but it won't prevent module code from 
modifying it. I really want to make sure it is not being mutated. 
So I'm now leaning toward this approach:

struct S {
   const(T) *rc;
}

so I won't have the copy/postblit issues. Doesn't this bypass the 
problems with `const(T) t`. The new risk is that somehow that 
member variable is initialized with a stack variable.

Thanks,
Dan
Nov 01 2013
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, November 01, 2013 14:28:55 Daniel Davidson wrote:

 On Thursday, 31 October 2013 at 19:39:44 UTC, Jonathan M Davis
 
 wrote:

 const and postblit fundamentally don't mix, because for it to
 work, you have
 to violate the type system. With postblits, the struct gets
 memcpied and then
 the postblit constructor has the chance to mutate the resulting
 object to make
 the pieces different that need to be different. However, to do
 that mutation
 when the object is const would mean mutating a const object
 which violates the
 type system. What we really need is copy constructors, but they
 haven't been
 added yet. At one point, Andrei was saying that he and Walter
 had a solution,
 but he didn't elaborate on it. I assume that it involved
 introducing copy
 constructors, but I don't know, and this issue has not yet been
 resolved.


 
 This makes sense, but I think we could have a simpler solution
 that does not involve a copy constructor addition. It seems the
 problem is postblit offers a hole in the system whereby
 developers could change immutable data. Because of that postblit
 and const don't mix. The simple fix would be take the
 implementation of postblit away from the developer. It seems the
 use case for postblit is really only to provide deep copy
 semantics by having developer throw in a bunch of dups where
 desired.


Deep copying is not the only reason to have a postblit. Smart pointers such as 
std.typecons.RefCounted require a postblit (or copy constructor if we had 
those) to do their job. Also, even if all you want is a deep copy, it's 
possible that the member being deep copied was allocated via something like 
malloc, in which case, the compiler couldn't take care of it for you.


 I am now convinced avoiding `const(T) t` as a member is wise
 advice. The suggestion of leaving it non-const and providing a
 const accessor is good - but it won't prevent module code from
 modifying it. I really want to make sure it is not being mutated.
 So I'm now leaning toward this approach:
 
 struct S {
 const(T) *rc;
 }
 
 so I won't have the copy/postblit issues. Doesn't this bypass the
 problems with `const(T) t`. The new risk is that somehow that
 member variable is initialized with a stack variable.


If you do that, then rc can be mutated (and so the struct can be mutated), but 
what it points to can't be. So, from the sounds if it, it does what you want. 
Though honestly, I wouldn't worry about the rest of a module mutating your 
member variable just because it has access to it. If you simply name member 
variables differently than the public API (e.g _rc instead of rc) and then 
provide a const property to access the member, then the odds are very low that 
you're going to accidentally access the member directly. And if you can't keep 
track of what's in a single module well enough to avoid this sort of problem, 
then your module is probably too big. But if you're paranoid, you can 
certainly add an extra layer indirection like you're suggesting. I wouldn't 
bother though.

- Jonathan M Davis
Nov 01 2013
parent "Daniel Davidson" <nospam spam.com> writes:
On Friday, 1 November 2013 at 20:29:54 UTC, Jonathan M Davis 
wrote:

 On Friday, November 01, 2013 14:28:55 Daniel Davidson wrote:

 On Thursday, 31 October 2013 at 19:39:44 UTC, Jonathan M Davis





 Deep copying is not the only reason to have a postblit. Smart 
 pointers such as
 std.typecons.RefCounted require a postblit (or copy constructor 
 if we had
 those) to do their job. Also, even if all you want is a deep 
 copy, it's
 possible that the member being deep copied was allocated via 
 something like
 malloc, in which case, the compiler couldn't take care of it 
 for you.


So, deepCopy is a reason for postblit and I probably common for 
those not wanting shared state. You have mentioned two more uses 
for postblit, ref counting and dealing with something allocated 
with malloc. Honestly I am not sure the impact of the latter. Are 
you referring to `struct T { S *s }` where the s was allocated 
with malloc?  I think you give up on const/immutable when ref 
counting anyway? Regardless, I imagine both of those cases are on 
the less common side and are probably not too troubled by const 
issues. I'm not suggesting getting rid of postblit. Neither am I 
against constructors. It just sounds like adding constructors is 
overkill if support for immutable language generated postblit 
were available. And I think it would be easy to add.

I'm just suggesting add an annotation that marks fields to be 
duped by a language generated postblit. The developer would not 
need to implement the postblit and it would be clear when/where 
sharing occurs. Then the language is still guaranteed immutable 
correct when using the language generated postblit.


 I am now convinced avoiding `const(T) t` as a member is wise
 advice. The suggestion of leaving it non-const and providing a
 const accessor is good - but it won't prevent module code from
 modifying it. I really want to make sure it is not being 
 mutated.
 So I'm now leaning toward this approach:
 
 struct S {
 const(T) *rc;
 }
 
 so I won't have the copy/postblit issues. Doesn't this bypass 
 the
 problems with `const(T) t`. The new risk is that somehow that
 member variable is initialized with a stack variable.


 If you do that, then rc can be mutated (and so the struct can 
 be mutated), but
 what it points to can't be. So, from the sounds if it, it does 
 what you want.
 Though honestly, I wouldn't worry about the rest of a module 
 mutating your
 member variable just because it has access to it.


Correct - I'm not worried about rc being modified as it is pretty 
easy to control that.

I was under the impression it is pretty easy to 
innocently/unknowingly mutate data. Suppose a member function 
passes that member `t` to any function in any other module. With 
slices and maps, what is really being passed around are pointers. 
Once you start passing them around in a non-const or 
non-immutable context you have objects sharing the same pieces 
data and eventually it could be changed, even though not in your 
module.


 If you simply name member
 variables differently than the public API (e.g _rc instead of 
 rc) and then
 provide a const property to access the member, then the odds 
 are very low that
 you're going to accidentally access the member directly. And if 
 you can't keep
 track of what's in a single module well enough to avoid this 
 sort of problem,
 then your module is probably too big.


Not sure I agree because passing non-const instances to functions 
anywhere in the system is now allowed if the member is non-const. 
So it has less to do with the size of your module and more to do 
with where you are passing it. If you are passing it around 
non-const the door is wide open.


 But if you're paranoid, you can
 certainly add an extra layer indirection like you're 
 suggesting. I wouldn't
 bother though.

 - Jonathan M Davis


Clearly I am paranoid :-)

But I think it is the case that const(T) and immutable(T) would 
be just fine and your reservations would be gone if the postblit 
were in fact immutable. If this were the case, wouldn't you 
change your stance and recommend const(T) or immutable(T) if in 
fact that is what the expectation is for the developer.

Thanks
Dan
Nov 06 2013