www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Temporary suspension of disbelief (invariant)

reply bearophile <bearophileHUGS lycos.com> writes:
I have not asked this in the D.learn newsgroup because I think it may be a bit
interesting for other general people too.

In D contract programming the invariant is not called before/after private
methods. I think that in some cases you may want to disable the invariant even
in some public methods. If your class/struct has a public method that
invalidates the state of the class instance, and one public method that fixes
the class instance state (I am thinking about certain data structures where in
a first phase you may add many items, and then you ask the data structure to
clean up itself and become coherent. This may avoid some computations), I have
found this way to implement it:



import std.stdio: writeln;

class Foo {
    private int someInstanceState = 1;

    // A ghost field, but it does't vanish in release mode
    //   as it is supposed to do:
    // http://d.puremagic.com/issues/show_bug.cgi?id=5027
    private int validState_ghostField = 0;

    private bool isInvariantEnabled() {
        return validState_ghostField >= 0;
    }

    private void incValidState()
        in {
            assert(validState_ghostField < validState_ghostField.max);
        } body {
            validState_ghostField++;
        }

    private void decValidState()
        in {
            assert(validState_ghostField > validState_ghostField.min);
        } body {
            validState_ghostField--;
        }

    invariant() {
        writeln("invariant validState_ghostField: ", validState_ghostField);
        if (isInvariantEnabled()) {
            writeln("Invariant is enabled");
            // tests the coherence of the instance state
            assert(someInstanceState == 1);
        } else {
            writeln("Invariant is disabled");
        }

    }

    public void fixState() {
        scope(exit) incValidState(); // scope(exit)?
        writeln("fixState validState_ghostField: ", validState_ghostField);
        someInstanceState = 1;
    }

    public void invalidateState() {
        scope(exit) decValidState(); // scope(exit)?
        writeln("invalidateState validState_ghostField: ",
validState_ghostField);
        someInstanceState = 0;
    }
}

void main() {
    auto f = new Foo;
    writeln("State 1: ", f.someInstanceState);
    writeln("isInvariantEnabled: ", f.isInvariantEnabled());
    f.invalidateState();
    writeln("State 2: ", f.someInstanceState);
    writeln("isInvariantEnabled: ", f.isInvariantEnabled());
    f.fixState();
    writeln("State 3: ", f.someInstanceState);
    writeln("isInvariantEnabled: ", f.isInvariantEnabled());
}


(Like the GC disable/enable validState_ghostField goes up and down, and allows
nesting too.)

All this looks bug-prone, and surely hairy, but it looks potentially useful. Is
it a good idea to design a class that uses such temporary suspension of the
invariant?

Bye,
bearophile
Oct 26 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
bearophile wrote:
 All this looks bug-prone, and surely hairy, but it looks potentially useful.
 Is it a good idea to design a class that uses such temporary suspension of
 the invariant?

An invariant that is not invariant is a meaningless attribute. It's like "logical constness" where classes claim to be const but aren't.
Oct 26 2010
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, October 26, 2010 17:25:20 Walter Bright wrote:
 bearophile wrote:
 All this looks bug-prone, and surely hairy, but it looks potentially
 useful. Is it a good idea to design a class that uses such temporary
 suspension of the invariant?

An invariant that is not invariant is a meaningless attribute. It's like "logical constness" where classes claim to be const but aren't.

The only case that I'm aware of where it makes sense for a public function to not run the invariant is for opAssign() to not run it before it itself is run ( http://d.puremagic.com/issues/show_bug.cgi?id=5058 ), but that's because opAssign() is replacing the state of the object rather than updating it, so you don't care what it's state was beforehand. That's a special case though. In the general case, it would overly complicate things to try and make an invariant not really be an invariant. If you really wanted to do that, you could use class variables which kept the state and made it so that the asserts in the invariant weren't run after a particular function call until another particular function call were made. But that strikes me as a colossally bad idea. Still, if you wanted such functionality, you _can_ get it, so I see no reason to complicate invariants just to make it easier to write what is likely bad code (or at least which is at a higher risk of being bad code). - Jonathan M Davis
Oct 26 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Jonathan M Davis wrote:
 In the general case, it would overly complicate things to try and make an 
 invariant not really be an invariant. If you really wanted to do that, you
could 
 use class variables which kept the state and made it so that the asserts in
the 
 invariant weren't run after a particular function call until another
particular 
 function call were made. But that strikes me as a colossally bad idea. Still,
if 
 you wanted such functionality, you _can_ get it, so I see no reason to 
 complicate invariants just to make it easier to write what is likely bad code 
 (or at least which is at a higher risk of being bad code).

Exactly. Having a class invariant that cheats is like painting over rust. It might look new, but the rust will shortly break through.
Oct 26 2010
prev sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 10/26/2010 07:25 PM, Walter Bright wrote:
 An invariant that is not invariant is a meaningless attribute. It's like
 "logical constness" where classes claim to be const but aren't.

an invariant which isn't used because it is too strict isn't much better. Example: class with some sort of state and public property function which grabs a bit out of some field or something inside method, state violates invariant and I can't call this property function I needed to use the property function, but I also needed the invariant so I shoehorned it into a regular function and explicitly made the calls. Ugly. Though I suppose it was a slightly different case than what bearophile is proposing
Oct 26 2010
prev sibling next sibling parent reply Rainer Deyke <rainerd eldwood.com> writes:
On 10/26/2010 18:00, bearophile wrote:
 All this looks bug-prone, and surely hairy, but it looks potentially
 useful. Is it a good idea to design a class that uses such temporary
 suspension of the invariant?

I think invariants should be checked whenever a public member function is called from outside the object. If setting the object to an invalid state from outside is valid, then clearly the state isn't really invalid. On the other hand, if the object itself calls it own public member functions, then no invariants should be checked. Not being able to call public member functions while the object is temporarily in an invalid state is too strict. This is a problem that I actually ran into while using D, and one of the reasons for why I stopped using invariants. -- Rainer Deyke - rainerd eldwood.com
Oct 26 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Rainer Deyke wrote:
 On the other hand, if the object itself calls it own public member
 functions, then no invariants should be checked.  Not being able to call
 public member functions while the object is temporarily in an invalid
 state is too strict.  This is a problem that I actually ran into while
 using D, and one of the reasons for why I stopped using invariants.

A solution is to redesign what the class considers public and private. A public member can be a shell around a private implementation, and other class members can call that private implementation without invoking the invariant.
Oct 26 2010
parent reply Rainer Deyke <rainerd eldwood.com> writes:
On 10/26/2010 20:16, Walter Bright wrote:
 Rainer Deyke wrote:
 On the other hand, if the object itself calls it own public member
 functions, then no invariants should be checked.  Not being able to call
 public member functions while the object is temporarily in an invalid
 state is too strict.  This is a problem that I actually ran into while
 using D, and one of the reasons for why I stopped using invariants.

A solution is to redesign what the class considers public and private. A public member can be a shell around a private implementation, and other class members can call that private implementation without invoking the invariant.

Writing wrapper functions is a waste of my time. Auto-generating wrapper functions through some sort of meta-programming magic is still a waste of my time, since the process cannot be completely automated. -- Rainer Deyke - rainerd eldwood.com
Oct 26 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Rainer Deyke wrote:
 On 10/26/2010 20:16, Walter Bright wrote:
 Rainer Deyke wrote:
 On the other hand, if the object itself calls it own public member
 functions, then no invariants should be checked.  Not being able to call
 public member functions while the object is temporarily in an invalid
 state is too strict.  This is a problem that I actually ran into while
 using D, and one of the reasons for why I stopped using invariants.

public member can be a shell around a private implementation, and other class members can call that private implementation without invoking the invariant.

Writing wrapper functions is a waste of my time. Auto-generating wrapper functions through some sort of meta-programming magic is still a waste of my time, since the process cannot be completely automated.

Then I agree that not using invariants at all is probably your best option.
Oct 26 2010
prev sibling next sibling parent reply Stanislav Blinov <blinov loniir.ru> writes:
  27.10.2010 4:00, bearophile wrote:
 I have not asked this in the D.learn newsgroup because I think it may be a bit
interesting for other general people too.

 In D contract programming the invariant is not called before/after private
methods. I think that in some cases you may want to disable the invariant even
in some public methods. If your class/struct has a public method that
invalidates the state of the class instance, and one public method that fixes
the class instance state (I am thinking about certain data structures where in
a first phase you may add many items, and then you ask the data structure to
clean up itself and become coherent. This may avoid some computations), I have
found this way to implement it:

several general logical states: e.g. "initializing" and "coherent". Given this, I don't see why you'd want to disable invariant checks rather than modify those checks instead to validate current logical state. In fact, that "ghost" field simply serves as a flag for invariant showing which logical state it should enforce. The fact that states are 'logical', e.g. are different while represented by the same physical fields doesn't always rule them out as, uh, class states: you could as well have two separate inner classes that perform initialization and polishing, each with its own invariant. Then you could use those inner classes' private methods (without causing their invariant checks), but in main class' invariant perform an assert on them to ensure their state is valid. IMHO, proper invariants should be strict and act by the numbers at all times, otherwise there's little to no gain in using them at all.
Oct 27 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Walter:

 An invariant that is not invariant is a meaningless attribute. It's like
 "logical constness" where classes claim to be const but aren't.
 
 It's not an invariant if it only works some of the time.
 
 It's like C++ and the mutable keyword, which allows an escape from the const
 rules. Which makes C++ const fairly useless.
 
 A person should be able to see the invariant in a class declaration, and know
 that it offers a guarantee. He should not be required to read over everything
 else in the source code looking for the absence of a method that disables it.

In this thread I have read many answers and comments, but I am not experienced enough yet about invariants and DbC to be able to answer them, I will need more time and more experiments. D after all is the first language where I have used DbC. I agree that an invariant that you may disable sounds like an oxymoron, and probably it leads to hairy code at best, or even to bug-prone code. Regarding "logical constness" see below my comments about the good answer by Stanislav Blinov. ---------------------- Stanislav Blinov:
 If I get this right, then it is by design that your class may have
 several general logical states: e.g. "initializing" and "coherent".

Right. An example from the Clojure language (it's a recent Lisp-like language that runs on the JavaVM). In Clojure most data is immutable. People that use Clojure have seen that this is nice and safe, but sometimes on the JavaVM this leads to low efficiency code especially when you create a data structure. So they have solved this problem inventing "transients", it's an initial phase when a data structure is being built, and it's mutable. Then when it's done it becomes immutable. This often regains lot of the lost performance introduced by immutables on the JVM. At the same way, I think that in some situations I may want to quickly build a data structure, in this phase the data structure may be internally not orderered (like an unbalanced tree), so in this phase the invariant may be false and it's better to disable it. (I think this is an uncommon situation, in most cases there is probably a way around this problem). It's like the transients for the immutability in Clojure.
 Given this, I don't see why you'd want to disable invariant checks
 rather than modify those checks instead to validate current logical
 state. In fact, that "ghost" field simply serves as a flag for invariant
 showing which logical state it should enforce. The fact that states are
 'logical', e.g. are different while represented by the same physical
 fields doesn't always rule them out as, uh, class states:

I think you are right.
 you could as
 well have two separate inner classes that perform initialization and
 polishing, each with its own invariant. Then you could use those inner
 classes' private methods (without causing their invariant checks), but
 in main class' invariant perform an assert on them to ensure their state
 is valid.

I see. I have never used this compound solution for the invariants. I will try it (as inner ones I may use structs instead, structs too support invariants in D), thank you for the idea :-) Thank you for all the interesting answers in this thread, I am learning. Bye, bearophile
Oct 27 2010
prev sibling next sibling parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
An issue I encountered in D1 with invariant is using delete: I have a  
method that deletes the current object, and obviously then any  
invariant would fail.
It would be nice to have a way to disable invariant in that case.
In D2, as delete is not allowed anymore (if I got it correctly) this  
is not a problem.

Fawzi
On 27-ott-10, at 15:08, Stanislav Blinov wrote:

 27.10.2010 4:00, bearophile wrote:
 I have not asked this in the D.learn newsgroup because I think it  
 may be a bit interesting for other general people too.

 In D contract programming the invariant is not called before/after  
 private methods. I think that in some cases you may want to disable  
 the invariant even in some public methods. If your class/struct has  
 a public method that invalidates the state of the class instance,  
 and one public method that fixes the class instance state (I am  
 thinking about certain data structures where in a first phase you  
 may add many items, and then you ask the data structure to clean up  
 itself and become coherent. This may avoid some computations), I  
 have found this way to implement it:

several general logical states: e.g. "initializing" and "coherent". Given this, I don't see why you'd want to disable invariant checks rather than modify those checks instead to validate current logical state. In fact, that "ghost" field simply serves as a flag for invariant showing which logical state it should enforce. The fact that states are 'logical', e.g. are different while represented by the same physical fields doesn't always rule them out as, uh, class states: you could as well have two separate inner classes that perform initialization and polishing, each with its own invariant. Then you could use those inner classes' private methods (without causing their invariant checks), but in main class' invariant perform an assert on them to ensure their state is valid. IMHO, proper invariants should be strict and act by the numbers at all times, otherwise there's little to no gain in using them at all.

Oct 27 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:
 An issue I encountered in D1 with invariant is using delete: I have a 
 method that deletes the current object, and obviously then any invariant 
 would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this is 
 not a problem.

It's not an invariant if it only works some of the time. It's like C++ and the mutable keyword, which allows an escape from the const rules. Which makes C++ const fairly useless. A person should be able to see the invariant in a class declaration, and know that it offers a guarantee. He should not be required to read over everything else in the source code looking for the absence of a method that disables it.
Oct 27 2010
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:
 well for methods that destroy the object I think that not checking the 
 invariant is something reasonable, and not a hole that destroys the 
 whole system. Being able to write methods that invalidate the object can 
 be useful IMHO.
 At the moment in those objects I simply have no invariant, and by the 
 way, no I don't think that having the invalidating methods always 
 external is really a good solution.
 You can do without, but sometime an  invalidate property (or similar) 
 has its uses.

I agree that having a method that destroys an object can be useful. Such classes can't have an invariant, however. The problem with an invalidate property is the user has to go through every declaration in the class looking for such a property (grep isn't sufficient because it can pick up false positives (nested functions) and can miss things inserted by templates and mixins). An invariant should be a guarantee, not "this is guaranteed unless someone somewhere violates it, which is perfectly legal". It's the same problem that C++ "mutable" produces, and it's not worth it.
Oct 27 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:
 the only thing is that the invariant is not checked *after* those method 
 that destroy the object, it will be ok to require a special name for 
 that method (for example destroy, dealloc or similar) if you want to 
 avoid misusage.

There is a method that does that, it's called the destructor. An invariant is not checked after the destructor is run.
Oct 27 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Fawzi Mohamed wrote:
 Not only that it might be useful for some methods to delete the object 
 itself.

The pimpl pattern should be used for such, if you want to use an invariant. Note that the C++0x shared_ptr uses pimpl in that the ref count is held separately from the object.
Oct 27 2010
prev sibling parent Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 22:41, Walter Bright wrote:

 Fawzi Mohamed wrote:
 the only thing is that the invariant is not checked *after* those  
 method that destroy the object, it will be ok to require a special  
 name for that method (for example destroy, dealloc or similar) if  
 you want to avoid misusage.

There is a method that does that, it's called the destructor. An invariant is not checked after the destructor is run.

ok I don't think that this is a major feature, or anything like that, so I will stop arguing after this, but I still believe that it would be useful. The destructor is an example of a method that should not check the invariant, but it does not address my use case, because one cannot explicitly call a destructor, and there is an important difference between a destructor called by the GC and a destructor called explicitly: the one called explicitly can access the memory of the object. Not only that it might be useful for some methods to delete the object itself. A classical example is reference counting, the last release (the one that sets the refCount to 0) should be able to delete the object. It is not natural to make this impossible, or to force al reference counted object to have no invariant.
Oct 27 2010
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, October 27, 2010 07:33:58 Fawzi Mohamed wrote:
 An issue I encountered in D1 with invariant is using delete: I have a
 method that deletes the current object, and obviously then any
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this
 is not a problem.

Except that calling a function on a deleted object would not be desirable, so having the invariant fail at that point would be a _good_ thing. If anything, you'd want _all_ function calls on a deleted object to fail invariant or no invariant, because none of them should be happening in the first place. clear() in D2 does (or at least did - I don't know what it's current state is) put an object in its default state prior to any constructor call which would likely violate any invariant, which, on the whole, is a good thing as well. However, there was discussion of making a cleared object have a nuked vtbl which would be even better, since then all function calls on it would fail period. - Jonathan M Davis
Oct 27 2010
prev sibling next sibling parent reply Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 18:59, Jonathan M Davis wrote:

 On Wednesday, October 27, 2010 07:33:58 Fawzi Mohamed wrote:
 An issue I encountered in D1 with invariant is using delete: I have a
 method that deletes the current object, and obviously then any
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this
 is not a problem.

Except that calling a function on a deleted object would not be desirable, so having the invariant fail at that point would be a _good_ thing. If anything, you'd want _all_ function calls on a deleted object to fail invariant or no invariant, because none of them should be happening in the first place. clear() in D2 does (or at least did - I don't know what it's current state is) put an object in its default state prior to any constructor call which would likely violate any invariant, which, on the whole, is a good thing as well. However, there was discussion of making a cleared object have a nuked vtbl which would be even better, since then all function calls on it would fail period. - Jonathan M Davis

call the invariant... *that* is what I was talking about, :) Fawzi
Oct 27 2010
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
Fawzi Mohamed wrote:
 
 On 27-ott-10, at 18:59, Jonathan M Davis wrote:
 
 On Wednesday, October 27, 2010 07:33:58 Fawzi Mohamed wrote:
 An issue I encountered in D1 with invariant is using delete: I have a
 method that deletes the current object, and obviously then any
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly) this
 is not a problem.

Except that calling a function on a deleted object would not be desirable, so having the invariant fail at that point would be a _good_ thing. If anything, you'd want _all_ function calls on a deleted object to fail invariant or no invariant, because none of them should be happening in the first place. clear() in D2 does (or at least did - I don't know what it's current state is) put an object in its default state prior to any constructor call which would likely violate any invariant, which, on the whole, is a good thing as well. However, there was discussion of making a cleared object have a nuked vtbl which would be even better, since then all function calls on it would fail period. - Jonathan M Davis

call the invariant... *that* is what I was talking about, :) Fawzi

What for? It doesn't now, and it shouldn't (and most probably will not) when it's design is solid and it's properly implemented. The purpose of clear() is: 1) call dtor (which implies checking invariant *only before* destruction), if object has it. 2) put an object in an invalid (from the compiler/runtime point of view) state: this has been discussed at length, and latest discussion ended with what Jonathan M Davis already mentioned: nuking object's virtual call table. 3) Leave object's memory alone so GC will later decide what to do with it If I haven't missed something, this is pretty much it. It's not a method, it's a function -> no invariant checks after call.
Oct 27 2010
prev sibling next sibling parent Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 20:20, Walter Bright wrote:

 Fawzi Mohamed wrote:
 An issue I encountered in D1 with invariant is using delete: I have  
 a method that deletes the current object, and obviously then any  
 invariant would fail.
 It would be nice to have a way to disable invariant in that case.
 In D2, as delete is not allowed anymore (if I got it correctly)  
 this is not a problem.

It's not an invariant if it only works some of the time. It's like C++ and the mutable keyword, which allows an escape from the const rules. Which makes C++ const fairly useless. A person should be able to see the invariant in a class declaration, and know that it offers a guarantee. He should not be required to read over everything else in the source code looking for the absence of a method that disables it.

well for methods that destroy the object I think that not checking the invariant is something reasonable, and not a hole that destroys the whole system. Being able to write methods that invalidate the object can be useful IMHO. At the moment in those objects I simply have no invariant, and by the way, no I don't think that having the invalidating methods always external is really a good solution. You can do without, but sometime an invalidate property (or similar) has its uses. Fawzi
Oct 27 2010
prev sibling parent Fawzi Mohamed <fawzi gmx.ch> writes:
On 27-ott-10, at 20:57, Walter Bright wrote:

 Fawzi Mohamed wrote:
 well for methods that destroy the object I think that not checking  
 the invariant is something reasonable, and not a hole that destroys  
 the whole system. Being able to write methods that invalidate the  
 object can be useful IMHO.
 At the moment in those objects I simply have no invariant, and by  
 the way, no I don't think that having the invalidating methods  
 always external is really a good solution.
 You can do without, but sometime an  invalidate property (or  
 similar) has its uses.

I agree that having a method that destroys an object can be useful. Such classes can't have an invariant, however. The problem with an invalidate property is the user has to go through every declaration in the class looking for such a property (grep isn't sufficient because it can pick up false positives (nested functions) and can miss things inserted by templates and mixins). An invariant should be a guarantee, not "this is guaranteed unless someone somewhere violates it, which is perfectly legal". It's the same problem that C++ "mutable" produces, and it's not worth it.

the only thing is that the invariant is not checked *after* those method that destroy the object, it will be ok to require a special name for that method (for example destroy, dealloc or similar) if you want to avoid misusage. The next method called on the object would fail (as it should), I often store invalid info on purpose before deleting to increase the probability of detecting such usages. invariant is still valid (and checked) before and after all methods. It is even impossible to call twice the destroy method (as the check at the beginning of it would fail. So I think that everything would stay perfectly well defined. I think it is a different situation than mutable
Oct 27 2010