digitalmars.D - Re: enforce()?

Sorry for not answering before, I was quite busy (despite in the meantime I
have written few posts and bug reports).

Thank you to all the people that have answered in this thread, and expecially
Walter that has given the first answer that I have understood in the thread.

In the beginning I didn't like the enforce() but now I can see that it's meant
for a quite different purpose. I was also worried of possible negative
performance impact coming from its so widespread usage in Phobos.


Regarding the exception hierarchy I think I agree with Leandro Lucarella. A
deep and complex exception hierarchy can be negative and overkill, the opposite
extrema, that is having zero specialized exceptions in Phobos, is bad. Using
just single-exception enforce() is bad, there is a small number of well chosen
exception types (organized in a flat or mostly flat list) that are useful to
have.

In my dlibs1 I have defined a flat list of few exceptions that I used:

ArgumentException     
EmptyException        
ExceptionTemplate     
IndexException        
IOException           
KeyException          
MissingMethodException
OverflowException     
RangeException        
UncomparableException 


Inside Phobos2 I have counted about 160 usages of the "body" keyword. I think
contract programming can be used more often inside Phobos2 (and maybe some
usages of enforce() can be turned into contract programming because they are
more similar to program sanity checks).

Bye,
bearophile

Jun 19 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


Walter and I discussed this and concluded that Phobos should handle its 
parameters as user input. Therefore they need to be scrubbed with hard 
tests, not contracts.

Andrei

Jun 19 2010

Lutger <lutger.blijdestijn gmail.com> writes:

Andrei Alexandrescu wrote:

 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 Walter and I discussed this and concluded that Phobos should handle its
 parameters as user input. Therefore they need to be scrubbed with hard
 tests, not contracts.
 
 Andrei


That is sensible. Are private functions (those only called from within Phobos) 
excluded from this rule?

Jun 19 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 06/19/2010 05:01 PM, Lutger wrote:
 Andrei Alexandrescu wrote:

 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 Walter and I discussed this and concluded that Phobos should handle its
 parameters as user input. Therefore they need to be scrubbed with hard
 tests, not contracts.

 Andrei


 That is sensible. Are private functions (those only called from within Phobos)
 excluded from this rule?


Yes, precisely. (The actual code does not fully obey this intention; 
there are contracts in places where there shouldn't be.)

Andrei

Jun 19 2010

Walter Bright <newshound1 digitalmars.com> writes:

Andrei Alexandrescu wrote:
 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 Walter and I discussed this and concluded that Phobos should handle its 
 parameters as user input. Therefore they need to be scrubbed with hard 
 tests, not contracts.


I should add that any library that may be used as a dll should have its 
interface API checked with hard tests, not contracts. This is because a dll 
cannot control who connects to it, and therefore must regard anything sent to
it 
as unvalidated user input.

Jun 19 2010

Walter Bright <newshound2 digitalmars.com> writes:

Vladimir Panteleev wrote:
 On Sun, 20 Jun 2010 03:04:31 +0300, Walter Bright 
 <newshound1 digitalmars.com> wrote:
 
 Andrei Alexandrescu wrote:
 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 its parameters as user input. Therefore they need to be scrubbed with 
 hard tests, not contracts.


 I should add that any library that may be used as a dll should have 
 its interface API checked with hard tests, not contracts. This is 
 because a dll cannot control who connects to it, and therefore must 
 regard anything sent to it as unvalidated user input.


 I don't see the logic in this...
 
 Are we talking about validating user input for the sake of security, or 
 debugging (catching bugs early)?


An input to a dll is user input, and should be validated (for the sake of 
security, and other reasons). Validating it is not debugging.


 If it's for the sake of debugging, debug checks should remain in debug 
 builds (that's what contracts are for?). Otherwise, you are stripping 
 the programmer of the choice between higher performance or more debug 
 checks.
 
 If it's for the sake of security - parameter validation in DLLs is 
 pointless. If you are able to load and call code from inside a DLL, you 
 are already able to do everything that the DLL can. DLLs don't have any 
 "setuid"-like properties. If we were talking, for example, about 
 syscalls for a kernel module (functions called from userland but 
 executed in kernel land), then that would be a completely different 
 situation.


If you, for example, provide a pluggable interface to your browser app, that's 
done using a dll, and you'd better validate anything you get through that
plugin 
interface!

 Also, I don't think that one rule can apply for everyone. For example, a 
 high-performance DLL may specify in its documentation that the function 
 parameters are not checked by the DLL and must be valid, otherwise 
 undefined behavior will occur. (I believe some Windows APIs do not check 
 some parameters and will cause access violations when called with 
 invalid parameters.)


If you don't validate the input, then you must accept the risk.

Jun 20 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 06/20/2010 06:18 PM, Vladimir Panteleev wrote:
 On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright
 <newshound2 digitalmars.com> wrote:

 An input to a dll is user input, and should be validated (for the sake
 of security, and other reasons). Validating it is not debugging.


 I don't understand why you're saying this. Security checks in DLL
 functions are pointless, for the reasons I already outlined:



I think the matter is simpler than that. Essentially DbC is modular 
integrity checking. If Phobos enforce()s parameters in calls, then it 
considers its own integrity a different matter than the integrity of the 
application it's used with.

If Phobos used contracts to validate parameters, it would directly share 
responsibility for the integrity of the entire application. That way, 
users will not be sure whether the failure is a bug in Phobos or one in 
their own code.


Andrei

Jun 20 2010

Walter Bright <newshound2 digitalmars.com> writes:

Vladimir Panteleev wrote:
 On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright 
 <newshound2 digitalmars.com> wrote:
 
 An input to a dll is user input, and should be validated (for the sake 
 of security, and other reasons). Validating it is not debugging.


 I don't understand why you're saying this. Security checks in DLL 
 functions are pointless, for the reasons I already outlined:


It's true that whenever user code is executed, that code can do anything.
Hello, 
ActiveX. But I still think it's sound practice to treat any data received from 
another program as untrusted, and validate it. Security, like I said, is only 
one reason. Another is to prevent bugs in external code from trashing your
process.

Jun 20 2010

Walter Bright <newshound2 digitalmars.com> writes:

Leandro Lucarella wrote:
 Why will you assume I'm so dumb that I won't use your
 interface correctly?


Windows has had major legacy compatibility issues because critical third party 
applications misused the APIs.

People *will* misuse your API, and you will get blamed for it. It's unfair, but 
that's how it works.

Jun 20 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 06/20/2010 11:08 PM, Leandro Lucarella wrote:
 Walter Bright, el 20 de junio a las 19:32 me escribiste:
 Leandro Lucarella wrote:
 Why will you assume I'm so dumb that I won't use your
 interface correctly?


 Windows has had major legacy compatibility issues because critical
 third party applications misused the APIs.

 People *will* misuse your API, and you will get blamed for it. It's
 unfair, but that's how it works.


 Luckily I haven't used Windows for about 10 years now =)

 It's really a shame that D will take the stupidity route.

 PS: I don't know how windows work, but if calling the Windows API is
      like going into kernel mode, and you can mess other processes, it
      seems reasonable to do check every API call as if it were user
      input, but if you're confined to your process, is really stupid.


Why is it stupid?

Andrei

Jun 21 2010

Sean Kelly <sean invisibleduck.org> writes:

Andrei Alexandrescu Wrote:

 On 06/20/2010 06:18 PM, Vladimir Panteleev wrote:
 On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright
 <newshound2 digitalmars.com> wrote:

 An input to a dll is user input, and should be validated (for the sake
 of security, and other reasons). Validating it is not debugging.


 I don't understand why you're saying this. Security checks in DLL
 functions are pointless, for the reasons I already outlined:


 
 I think the matter is simpler than that. Essentially DbC is modular 
 integrity checking. If Phobos enforce()s parameters in calls, then it 
 considers its own integrity a different matter than the integrity of the 
 application it's used with.
 
 If Phobos used contracts to validate parameters, it would directly share 
 responsibility for the integrity of the entire application. That way, 
 users will not be sure whether the failure is a bug in Phobos or one in 
 their own code.


If a unrecoverable failure occurs within the process, does it matter where the
error originated?

I've been thinking about this a bit and am starting to wonder about the benefit
of distinguishing API boundary integrity checking vs. internal integrity
checking.  First, what if a library eats its own dogfood?  If my library
provides a public method to spawn threads and the library itself uses threads
internally then I have two different methods of checking the integrity of my
own library, each possibly throwing different exceptions (enforce throws
Exception while assert throws AssertError).  At the very least, this seems like
it could cause maintenance issues because a logic change deep within the
library may require additional exception handling to deal with what are
intended to be user-facing errors.

In a similar vein, if contracts are used within an API but a different mode of
checking is used at the API boundary, when contracts are enabled in that API
the user is faced with the bizarre issue of receiving AssertErrors from
internal API logic errors but only Exceptions from his own logic errors for API
boundary calls.  When you say that DbC is modular I'd presume that means it's
encapsulated within each distinct subsystem, but it seems like this isn't true
at all.  The alternative is to turn DbC off in the library and either live with
undefined behavior or a hard crash if there's a bug in the library, a
circumstance which is again forced upon the user.

Regarding DbC, I can't say that I've ever worked on a system where lives hung
in the balance (an admittedly extreme example of where DbC is useful), but if I
were sufficiently concerned about process integrity that I had contracts
enabled then I don't think I would trust that a third-party library was
bug-free and therefore didn't need its own contract checking in place.  Once
I've accepted the cost of integrity checking I want it everywhere, not just in
my own code.  It makes for consistent error checking behavior (I'd assume there
is a system in place to trap DbC errors specifically) and provides full-process
integrity checking.

I think the only boundary that really matters is the process boundary.  Any
error within a process has the same effect regardless of whether it's in user
code or library code--corrupted memory, segfaults, etc--so why make a
distinction between code I wrote and code someone else wrote?  By the same
token, if the user chooses to disable contracts then why force them upon him
for some errors but not others?  The user is making the explicit choice to run
his process through a meat-grinder if something unexpected happens, he's
effectively asserting that his code is perfect, so why tell him that "no, it's
actually not."

For me, the more difficult issue is how much integrity checking should be done.
 For example, I created an AVL tree a while back that verified that the tree
was still properly balanced after every API call.  This was great from a code
verification standpoint, but the check completely violated the complexity
guarantees and ultimately checked something that could have been proven to a
reasonable degree of confidence through code reviews and unit testing.  Should
checks like this be enabled automatically when DbC is turned on?  Should there
be different levels of DbC?  In some respects I feel like there's a difference
between in/out contracts and invariants, but even that doesn't seem completely
right.  Thoughts?

Jun 21 2010

bearophile <bearophileHUGS lycos.com> writes:

Sean Kelly:

First, what if a library eats its own dogfood? If my library provides a public
method to spawn threads and the library itself uses threads internally then I
have two different methods of checking the integrity of my own library, each
possibly throwing different exceptions (enforce throws Exception while assert
throws AssertError).<

I think that Design by contract, to be useful, needs to be embraced. You need
to trust it and use it everywhere. Now I use DbC quite often in my D code and I
appreciate it. The DbC feature I miss mostly is the "old" (view of the original
input data).

For example, I created an AVL tree a while back that verified that the tree was
still properly balanced after every API call. This was great from a code
verification standpoint, but the check completely violated the complexity
guarantees and ultimately checked something that could have been proven to a
reasonable degree of confidence through code reviews and unit testing. Should
checks like this be enabled automatically when DbC is turned on? Should there
be different levels of DbC? In some respects I feel like there's a difference
between in/out contracts and invariants, but even that doesn't seem completely
right. Thoughts?<

Using Design by Contract is not easy, you need to train yourself to use it
well. A problem is that DbC is uncommon, only Eiffel and few other languages
use it seriously, so lot of D users have to learn DbC on D itself.

I face your problem putting inside the contracts code that doesn't change the
complexity of the code it guards (so for example if the code is O(n^2) I don't
add contracts that perform O(n^3) computations).

Then where it's useful I add stronger tests (that can be slower) inside
debug{}. You can see it here too:
http://www.digitalmars.com/pnews/read.php?server=news.digitalmars.com&group=digitalmars.D&artnum=109395&header

Inside the invariant there is O(1) test code, and it also contains inside a
debug{} O(n) test code (that is .

In debug mode I want to test the code very well, while in normal nonrelease
mode I can accept less stringent tests that make the code usable.

Unittests and DbC (and integration tests, functional tests, smoke tests, etc)
are both useful, they do different things :-) For example an unittest can tell
me a function is wrong, but a loop invariant can tell me where the bug is and
when it happens inside the function :-)

Bye,
bearophile

Jun 21 2010

Walter Bright <newshound2 digitalmars.com> writes:

Sean Kelly wrote:
 Regarding DbC, I can't say that I've ever worked on a system where lives hung
 in the balance (an admittedly extreme example of where DbC is useful),


I have, and here's how it's done:

http://www.drdobbs.com/blog/archives/2009/10/safe_systems_fr.html

http://www.drdobbs.com/blog/archives/2009/11/designing_safe.html

I really wish this was more widely known in the software engineering business. 
It's frustrating to see it relearned the hard way, over and over.

And not just the software business, I saw a technical overview of the BP oil 
spill failure, and the rig design violated just about every principle of safe 
system design.

Jun 22 2010

Sean Kelly <sean invisibleduck.org> writes:

Walter Bright Wrote:

 Sean Kelly wrote:
 Regarding DbC, I can't say that I've ever worked on a system where lives hung
 in the balance (an admittedly extreme example of where DbC is useful),


 I have, and here's how it's done:
 
 http://www.drdobbs.com/blog/archives/2009/10/safe_systems_fr.html
 
 http://www.drdobbs.com/blog/archives/2009/11/designing_safe.html
 
 I really wish this was more widely known in the software engineering business. 
 It's frustrating to see it relearned the hard way, over and over.
 
 And not just the software business, I saw a technical overview of the BP oil 
 spill failure, and the rig design violated just about every principle of safe 
 system design.


A coworker of mine knows a guy who had workers on that rig and told me this
story the other day.  Apparently, there's a system on the drill that when a
failure occurs a cap drops over the hole and shears the drill.  The BP rig was
drilling unusually deep though, and as a result the drill had to be incredibly
hard.  For this and other reasons, the safety system was estimated to have a
70% failure rate.  Furthermore, the rig was known to be on the verge of
failure.  He implored the BP folks to shut it down, but they refused so in
desperation he hired people to fly his team off the rig, fearing for their
safety.  The rig failed a few hours after his team was evacuated.

While I've never worked on systems where lives hang in the balance, I have
worked on systems where 100% uptime is required.  I favor the Erlang approach
where a system is a web of interconnected, redundant processes that terminate
on errors.  I've found this design an extremely hard sell in the internet
server world though.  The design takes more planning and people are in too much
of a hurry.

Jun 22 2010

Sean Kelly <sean invisibleduck.org> writes:

Sean Kelly Wrote:
 
 While I've never worked on systems where lives hang in the balance, I have
worked on systems where 100% uptime is required.  I favor the Erlang approach
where a system is a web of interconnected, redundant processes that terminate
on errors.  I've found this design an extremely hard sell in the internet
server world though.  The design takes more planning and people are in too much
of a hurry.


I should add that I'm hoping the message passing model in D will help encourage
reliable system design.  With thread isolation it should be pretty easy to move
parts of a program into separate processes as need dictates.

Jun 22 2010

Lutger <lutger.blijdestijn gmail.com> writes:

Sean Kelly wrote:

 Sean Kelly Wrote:
 
 While I've never worked on systems where lives hang in the balance, I have
 worked on systems where 100% uptime is required.  I favor the Erlang approach
 where a system is a web of interconnected, redundant processes that terminate
 on errors.  I've found this design an extremely hard sell in the internet
 server world though.  The design takes more planning and people are in too
 much of a hurry.


 I should add that I'm hoping the message passing model in D will help
 encourage reliable system design.  With thread isolation it should be pretty
 easy to move parts of a program into separate processes as need dictates.


Can we look forward to seeing ipc supported in phobos via the same interface in 
the future? I really like the api you have created.

Jun 23 2010

Sean Kelly <sean invisibleduck.org> writes:

Lutger <lutger.blijdestijn gmail.com> wrote:
 Sean Kelly wrote:
 
 Sean Kelly Wrote:
 
 While I've never worked on systems where lives hang in the balance,
 I have
 worked on systems where 100% uptime is required.  I favor the Erlang
 approach
 where a system is a web of interconnected, redundant processes that
 terminate
 on errors.  I've found this design an extremely hard sell in the
 internet
 server world though.  The design takes more planning and people are
 in too
 much of a hurry.


 I should add that I'm hoping the message passing model in D will help
 encourage reliable system design.  With thread isolation it should be
 pretty
 easy to move parts of a program into separate processes as need
 dictates.


 Can we look forward to seeing ipc supported in phobos via the same
 interface in 
 the future? I really like the api you have created.


Yes. The core send/receive API should work just fine for IPC, and it's
definitely on the map.  The greatest obstacle there is probably the need
for a solid serialization/deserialization package in Phobos.

Jun 24 2010

Jacob Carlborg <doob me.com> writes:

On 2010-06-25 05:17, Sean Kelly wrote:
 Lutger<lutger.blijdestijn gmail.com>  wrote:
 Sean Kelly wrote:

 Sean Kelly Wrote:
 While I've never worked on systems where lives hang in the balance,
 I have
 worked on systems where 100% uptime is required.  I favor the Erlang
 approach
 where a system is a web of interconnected, redundant processes that
 terminate
 on errors.  I've found this design an extremely hard sell in the
 internet
 server world though.  The design takes more planning and people are
 in too
 much of a hurry.


 I should add that I'm hoping the message passing model in D will help
 encourage reliable system design.  With thread isolation it should be
 pretty
 easy to move parts of a program into separate processes as need
 dictates.


 Can we look forward to seeing ipc supported in phobos via the same
 interface in
 the future? I really like the api you have created.


 Yes. The core send/receive API should work just fine for IPC, and it's
 definitely on the map.  The greatest obstacle there is probably the need
 for a solid serialization/deserialization package in Phobos.


I have a serialization library, http://dsource.org/projects/orange/ , 
this is a list of some its features:

* It handles both serializing and deserializing
* It automatically serializes the base classes
* It supports events (before and after (de)serializing)
* It supports non-serialized fields (you can say that some fields in a 
class should not be serialized)
* It's licensed under the Boost license
* It's fairly std/runtime library independent
* You can create new archive types and use them with the existing serializer
* Serializes through base class references
* Serializes third party types

Currently it only works using Tango but the only part of the library 
that is dependent on the std/runtime library is XMLArchive, I'm 
currently working on porting it to Phobos. It also needs testing.

Also issue 2844 and the one, can't find it now, about getMembers is not 
implemented at all (returns an empty array).

-- 
/Jacob Carlborg

Jun 25 2010

Jacob Carlborg <doob me.com> writes:

On 2010-06-25 14:54, Jacob Carlborg wrote:
 On 2010-06-25 05:17, Sean Kelly wrote:
 Lutger<lutger.blijdestijn gmail.com> wrote:
 Sean Kelly wrote:

 Sean Kelly Wrote:
 While I've never worked on systems where lives hang in the balance,
 I have
 worked on systems where 100% uptime is required. I favor the Erlang
 approach
 where a system is a web of interconnected, redundant processes that
 terminate
 on errors. I've found this design an extremely hard sell in the
 internet
 server world though. The design takes more planning and people are
 in too
 much of a hurry.


 I should add that I'm hoping the message passing model in D will help
 encourage reliable system design. With thread isolation it should be
 pretty
 easy to move parts of a program into separate processes as need
 dictates.


 Can we look forward to seeing ipc supported in phobos via the same
 interface in
 the future? I really like the api you have created.


 Yes. The core send/receive API should work just fine for IPC, and it's
 definitely on the map. The greatest obstacle there is probably the need
 for a solid serialization/deserialization package in Phobos.


 I have a serialization library, http://dsource.org/projects/orange/ ,
 this is a list of some its features:

 * It handles both serializing and deserializing
 * It automatically serializes the base classes
 * It supports events (before and after (de)serializing)
 * It supports non-serialized fields (you can say that some fields in a
 class should not be serialized)
 * It's licensed under the Boost license
 * It's fairly std/runtime library independent
 * You can create new archive types and use them with the existing
 serializer
 * Serializes through base class references
 * Serializes third party types

 Currently it only works using Tango but the only part of the library
 that is dependent on the std/runtime library is XMLArchive, I'm
 currently working on porting it to Phobos. It also needs testing.

 Also issue 2844 and the one, can't find it now, about getMembers is not
 implemented at all (returns an empty array).


... should be fixed.


-- 
/Jacob Carlborg

Jun 25 2010

Jacob Carlborg <doob me.com> writes:

On 2010-06-25 17:40, Robert Jacques wrote:
 On Fri, 25 Jun 2010 08:54:54 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2010-06-25 05:17, Sean Kelly wrote:
 Lutger<lutger.blijdestijn gmail.com> wrote:
 Sean Kelly wrote:

 Sean Kelly Wrote:
 While I've never worked on systems where lives hang in the balance,
 I have
 worked on systems where 100% uptime is required. I favor the Erlang
 approach
 where a system is a web of interconnected, redundant processes that
 terminate
 on errors. I've found this design an extremely hard sell in the
 internet
 server world though. The design takes more planning and people are
 in too
 much of a hurry.


 I should add that I'm hoping the message passing model in D will help
 encourage reliable system design. With thread isolation it should be
 pretty
 easy to move parts of a program into separate processes as need
 dictates.


 Can we look forward to seeing ipc supported in phobos via the same
 interface in
 the future? I really like the api you have created.


 Yes. The core send/receive API should work just fine for IPC, and it's
 definitely on the map. The greatest obstacle there is probably the need
 for a solid serialization/deserialization package in Phobos.


 I have a serialization library, http://dsource.org/projects/orange/ ,
 this is a list of some its features:

 * It handles both serializing and deserializing
 * It automatically serializes the base classes
 * It supports events (before and after (de)serializing)
 * It supports non-serialized fields (you can say that some fields in a
 class should not be serialized)
 * It's licensed under the Boost license
 * It's fairly std/runtime library independent
 * You can create new archive types and use them with the existing
 serializer
 * Serializes through base class references
 * Serializes third party types

 Currently it only works using Tango but the only part of the library
 that is dependent on the std/runtime library is XMLArchive, I'm
 currently working on porting it to Phobos. It also needs testing.

 Also issue 2844 and the one, can't find it now, about getMembers is
 not implemented at all (returns an empty array).


 I'll volunteer to help test (and to add JSON capabilities) when you're
 ready.


It's ready to be tested with D1 and Tango. You can also start building a 
JSON archive, if you use D2 this will also make sure there is no other 
Tango dependencies (other than in XMLArchive). Let me no if you need any 
help.

-- 
/Jacob Carlborg

Jun 25 2010

"Robert Jacques" <sandford jhu.edu> writes:

On Fri, 25 Jun 2010 08:54:54 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2010-06-25 05:17, Sean Kelly wrote:
 Lutger<lutger.blijdestijn gmail.com>  wrote:
 Sean Kelly wrote:

 Sean Kelly Wrote:
 While I've never worked on systems where lives hang in the balance,
 I have
 worked on systems where 100% uptime is required.  I favor the Erlang
 approach
 where a system is a web of interconnected, redundant processes that
 terminate
 on errors.  I've found this design an extremely hard sell in the
 internet
 server world though.  The design takes more planning and people are
 in too
 much of a hurry.


 I should add that I'm hoping the message passing model in D will help
 encourage reliable system design.  With thread isolation it should be
 pretty
 easy to move parts of a program into separate processes as need
 dictates.


 Can we look forward to seeing ipc supported in phobos via the same
 interface in
 the future? I really like the api you have created.


 Yes. The core send/receive API should work just fine for IPC, and it's
 definitely on the map.  The greatest obstacle there is probably the need
 for a solid serialization/deserialization package in Phobos.


 I have a serialization library, http://dsource.org/projects/orange/ ,  
 this is a list of some its features:

 * It handles both serializing and deserializing
 * It automatically serializes the base classes
 * It supports events (before and after (de)serializing)
 * It supports non-serialized fields (you can say that some fields in a  
 class should not be serialized)
 * It's licensed under the Boost license
 * It's fairly std/runtime library independent
 * You can create new archive types and use them with the existing  
 serializer
 * Serializes through base class references
 * Serializes third party types

 Currently it only works using Tango but the only part of the library  
 that is dependent on the std/runtime library is XMLArchive, I'm  
 currently working on porting it to Phobos. It also needs testing.

 Also issue 2844 and the one, can't find it now, about getMembers is not  
 implemented at all (returns an empty array).


I'll volunteer to help test (and to add JSON capabilities) when you're  
ready.

Jun 25 2010

BCS <none anon.com> writes:

Hello Walter,

 And not just the software business, I saw a technical overview of the
 BP oil spill failure, and the rig design violated just about every
 principle of safe system design.


link by chance?

-- 
... <IXOYE><

Jun 22 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 06/21/2010 01:14 PM, Sean Kelly wrote:
 Andrei Alexandrescu Wrote:

 On 06/20/2010 06:18 PM, Vladimir Panteleev wrote:
 On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright
 <newshound2 digitalmars.com>  wrote:

 An input to a dll is user input, and should be validated (for
 the sake of security, and other reasons). Validating it is not
 debugging.


 I don't understand why you're saying this. Security checks in
 DLL functions are pointless, for the reasons I already outlined:



 I think the matter is simpler than that. Essentially DbC is
 modular integrity checking. If Phobos enforce()s parameters in
 calls, then it considers its own integrity a different matter than
 the integrity of the application it's used with.

 If Phobos used contracts to validate parameters, it would directly
 share responsibility for the integrity of the entire application.
 That way, users will not be sure whether the failure is a bug in
 Phobos or one in their own code.


 If a unrecoverable failure occurs within the process, does it matter
 where the error originated?


It does matter for the postmortem. Black box ftw.

 I've been thinking about this a bit and am starting to wonder about
 the benefit of distinguishing API boundary integrity checking vs.
 internal integrity checking.  First, what if a library eats its own
 dogfood?  If my library provides a public method to spawn threads and
 the library itself uses threads internally then I have two different
 methods of checking the integrity of my own library, each possibly
 throwing different exceptions (enforce throws Exception while assert
 throws AssertError).  At the very least, this seems like it could
 cause maintenance issues because a logic change deep within the
 library may require additional exception handling to deal with what
 are intended to be user-facing errors.


Any complex API will face at some point some tension between reusing and 
duplicating code. Per the classic joke:

"A mathematician was in a habit of making a cup of tea when working late 
at night. His normal method was to get the teapot from the cupboard, 
take the teapot to the sink, add water, heat to boiling, then make the 
cup of tea. Unfortunately, one night when he went to make tea, the 
teapot was already full of water and sitting on the stove. He thought 
about this for several minutes, then emptied the teapot and put it back 
in the cupboard, thereby reducing this to a previously solved problem."

It's often the case that an API throws some water for the sake of 
reusing itself.

 In a similar vein, if contracts are used within an API but a
 different mode of checking is used at the API boundary, when
 contracts are enabled in that API the user is faced with the bizarre
 issue of receiving AssertErrors from internal API logic errors but
 only Exceptions from his own logic errors for API boundary calls.
 When you say that DbC is modular I'd presume that means it's
 encapsulated within each distinct subsystem, but it seems like this
 isn't true at all.  The alternative is to turn DbC off in the library
 and either live with undefined behavior or a hard crash if there's a
 bug in the library, a circumstance which is again forced upon the
 user.


When I say DbC is modular I have in mind the following: "assert() inside 
a well-defined library entity (e.g. Phobos) is supposed to check the 
integrity of that entity, not the integrity of the entity using it."

I think that's reasonable. Integrity is a cross-cutting concern in 
memory-unsafe programs, but not in memory-safe programs. (Whoa, that's 
interesting.)

 Regarding DbC, I can't say that I've ever worked on a system where
 lives hung in the balance (an admittedly extreme example of where DbC
 is useful), but if I were sufficiently concerned about process
 integrity that I had contracts enabled then I don't think I would
 trust that a third-party library was bug-free and therefore didn't
 need its own contract checking in place.  Once I've accepted the cost
 of integrity checking I want it everywhere, not just in my own code.
 It makes for consistent error checking behavior (I'd assume there is
 a system in place to trap DbC errors specifically) and provides
 full-process integrity checking.


There's a large spectrum between "people will die" etc. and "I don't 
give a flying frak". I think an application under construction may well 
choose to use contracts for itself but not for the well-tested (ahem) 
Phobos.

 I think the only boundary that really matters is the process
 boundary.  Any error within a process has the same effect regardless
 of whether it's in user code or library code--corrupted memory,
 segfaults, etc--so why make a distinction between code I wrote and
 code someone else wrote?


For post-mortem and for assigning blame appropriately. If Phobos used 
DbC on user-passed inputs it would essentially share blame for the 
application integrity with all applications.

 By the same token, if the user chooses to
 disable contracts then why force them upon him for some errors but
 not others?  The user is making the explicit choice to run his
 process through a meat-grinder if something unexpected happens, he's
 effectively asserting that his code is perfect, so why tell him that
 "no, it's actually not."


That's a good point, but imho not enough to challenge the status quo. It 
might make sense to have a "unsafe" build for Phobos that assumes 
absolutely all arguments are correct. I wonder how much of an 
improvement it would bring.

 For me, the more difficult issue is how much integrity checking
 should be done.  For example, I created an AVL tree a while back that
 verified that the tree was still properly balanced after every API
 call.  This was great from a code verification standpoint, but the
 check completely violated the complexity guarantees and ultimately
 checked something that could have been proven to a reasonable degree
 of confidence through code reviews and unit testing.  Should checks
 like this be enabled automatically when DbC is turned on?  Should
 there be different levels of DbC?  In some respects I feel like
 there's a difference between in/out contracts and invariants, but
 even that doesn't seem completely right.  Thoughts?


I take no prisoners there: integrity checks MUST NOT affect complexity. 
Complexity is part of the spec, so such checks would automatically 
violate the spec.

At some point in its history, the binary search functions in 
std.algorithm had a enforce(isSorted) check. Should be somewhere buried 
in the svn history. It was my fault, and I didn't realize it until after 
I waited for one day next to a script to complete.


Andrei

Jun 27 2010

Adrian Matoga <epi atari8.info> writes:

Leandro Lucarella pisze:
 Walter Bright, el 20 de junio a las 19:32 me escribiste:
 Leandro Lucarella wrote:
 Why will you assume I'm so dumb that I won't use your
 interface correctly?


 third party applications misused the APIs.

 People *will* misuse your API, and you will get blamed for it. It's
 unfair, but that's how it works.


 Luckily I haven't used Windows for about 10 years now =)
 
 It's really a shame that D will take the stupidity route.
 
 PS: I don't know how windows work, but if calling the Windows API is
     like going into kernel mode, and you can mess other processes, it
     seems reasonable to do check every API call as if it were user
     input, but if you're confined to your process, is really stupid.
 


It was 15 years ago, at the times of 3.x and 95, when Windows behaved 
like that.

The problem applies not only to Windows, but any API you would imagine.
A common situation is when you need to do your job quickly using only 
some part of a library which otherwise you aren't going to study 
thoroughly, or you want only a proof-of-concept. And if your attempting 
to use something new to you, you do make mistakes, no matter how you are 
convinced yo do not.
If the API is defined not by documentation (which is often a tissue of 
lies, and hardly ever it's unambiguous), but by means of input checking, 
you have benefits in two fields: 1) developers of library had to think 
what they wanted to do, so library probably works, and it's less 
probable that its new versions will break the compatiblity, and 2) users 
of the library will be warned quickly, saving their time.

It's not about messing other processes. It's about saving your time, 
otherwise consumed by effects of common mistakes, misunderstanding the 
documentation, or working in a hurry. And your time costs much more than 
the time of bazillion argument checks.

Jun 21 2010

Sean Kelly <sean invisibleduck.org> writes:

Rory McGuire Wrote:
 
 I think perhaps you mis-understood, it is mostly not stupidity that causes  
 people to use
 undocumented "features" of an API but rather, it is people being overly  
 "clever".


Or sometimes simply desperation.  There are some classes of apps that require
the use of undocumented API calls to operate on Windows--I believe disk
defragmenters are one example.  Microsoft rightly didn't document these calls
because it wasn't prepared to support them long-term, but in doing so they also
prevented users from doing necessary work and effectively forced them into
using API calls that might change unexpectedly.  I think these users accept
this problem and do the necessary verification and updating when new OS
revisions are released however.

Jun 21 2010

Don <nospam nospam.com> writes:

Sean Kelly wrote:
 Rory McGuire Wrote:
 I think perhaps you mis-understood, it is mostly not stupidity that causes  
 people to use
 undocumented "features" of an API but rather, it is people being overly  
 "clever".


 Or sometimes simply desperation.  There are some classes of apps that require
the use of undocumented API calls to operate on Windows--I believe disk
defragmenters are one example.  Microsoft rightly didn't document these calls
because it wasn't prepared to support them long-term, but in doing so they also
prevented users from doing necessary work and effectively forced them into
using API calls that might change unexpectedly.  I think these users accept
this problem and do the necessary verification and updating when new OS
revisions are released however.


Remember Windows 3.0? File handling involved undocumented API calls!

Jun 21 2010

Norbert Nemec <Norbert Nemec-online.de> writes:

On 20/06/10 22:17, Walter Bright wrote:
 An input to a dll is user input, and should be validated (for the sake
 of security, and other reasons). Validating it is not debugging.


In that case, feel free to compile DLLs with external contract checking 
switched on, but please do not blur the conceptual distinction between 
contracts and exceptions.

You are talking about compiling a libary into a binary DLL that should 
be fit for general usage. In that case, there are good reasons to leave 
the input contract checking active.

In the general case, however, the library user has the control over how 
to compile the library and link to it (just think of inlining). In this 
case, the library user should be allowed to switch off the contract 
checking (at their own risk!)

Conceptually, the ultimate solution would certainly be to place code for 
input contract checking in the *calling* code. After all, this checking 
code serves to debug the calling code, so it should be left to the 
caller to decide whether checking is necessary.

This approach would also allow the compiler to optimize out some checks 
when their correctness can be tested at compile time.

Output contract checks, on the other hand should be compiled inside the 
returning routine.

After all, it is all a matter of trust. A language designer should trust 
the language user to know what he is doing. A library designer should 
trust the library user to act responsibly. After all - if the 
application breaks it is the application designer who has to answer for it.

Jun 28 2010

bearophile <bearophileHUGS lycos.com> writes:

Norbert Nemec:
 [...] to place code for input contract checking in the *calling* code. [...]
 Output contract checks, on the other hand should be compiled inside the 
 returning routine.


Is this a positive thing to do? Can this be done? (D must support separate
compilation, but in many situations this is not done/necessary, so maybe in
such situations it can be done). Is Eiffel doing it? if it's a good thing and
it's doable then what kind of changes does it require to the compiler?

Bye,
bearophile

Jun 28 2010

Norbert Nemec <Norbert Nemec-online.de> writes:

On 28/06/10 12:59, bearophile wrote:
 Norbert Nemec:
 [...] to place code for input contract checking in the *calling* code. [...]
 Output contract checks, on the other hand should be compiled inside the
 returning routine.


 Is this a positive thing to do? Can this be done? (D must support separate
compilation, but in many situations this is not done/necessary, so maybe in
such situations it can be done). Is Eiffel doing it? if it's a good thing and
it's doable then what kind of changes does it require to the compiler?


These are good and pragmatic questions that you ask.

The whole issue only arises when doing separate compilation of a library 
and an application. (I use the term "application" for any code that uses 
the library.)

In an idea world (beware, I am switching of "pragmatic thinking mode" 
for a moment), I would describe the situation would as follows:

Either part can be compiled in "debug" mode or in "release" mode. Debug 
mode in the library means that you want to debug the library code 
itself. Release mode in the library means that you trust the library 
code to be correct and switch off all internal checks.

The interesting situation is when you compile the application in debug 
mode, using a stable library compiled in release mode. In this case, the 
input contracts of the library need to be checked to catch bugs in the 
application. Once the testing phase is finished, you want to compile the 
application and get rid of the contract checks.

In this idealized picture, the input contracts should clearly be checked 
in the application code so that the application developer has control 
over them.

Now, for the real world: Code is hardly never 100% bug free. I agree. 
With this argument, however, the concept of debug and release mode 
becomes pointless as you should never switch off the checks anyway. 
Assuming that you are not quite as paranoid you might still reach a 
point where you trust your code to switch off the checks. How about 
input contracts of a library?

As Simen mentioned in his post, there is the issue of library authors 
trying to avoid blame from application authors and the other way around. 
Ultimately, this is exactly what contracts are for: formalize the 
interface as much as possible and make it machine-checkable. If an 
application breaks, compile it in debug mode and see whether it finds 
the problem. With input contracts checked in the calling code, this 
would automatically switch on all the relevant checks to identify bugs 
in the application. If the application designer still tries to blame the 
library, why not simply supply a library compiled in debug mode? Any 
violation of a contract or an assertion unquestionably signals a bug in 
the code.

Can it be done? Certainly. The contract is part of the interface, so 
rather than compiling it into the DLL, it should be stored with the 
interface definitions and left for the application compiler to insert if 
requested. The code for this should not be any more involved than 
function inlining. Effectively, every function with contracts would be 
turned into a wrapper that first checks the contracts and then calls the 
real function in the DLL. The application compiler could then simply 
decide whether to inline the wrapper or simply call the function without 
checks.

I have no idea how Eiffel does it, but I am quite certain that this 
solution is following the original spirit of DbC of Bertrand Meyer.

Greetings,
Norbert

Jun 30 2010

Sean Kelly <sean invisibleduck.org> writes:

Norbert Nemec Wrote:

 On 28/06/10 12:59, bearophile wrote:
 Norbert Nemec:
 [...] to place code for input contract checking in the *calling* code. [...]
 Output contract checks, on the other hand should be compiled inside the
 returning routine.


 Is this a positive thing to do? Can this be done? (D must support separate
compilation, but in many situations this is not done/necessary, so maybe in
such situations it can be done). Is Eiffel doing it? if it's a good thing and
it's doable then what kind of changes does it require to the compiler?


 These are good and pragmatic questions that you ask.
 
 The whole issue only arises when doing separate compilation of a library 
 and an application. (I use the term "application" for any code that uses 
 the library.)
 
 In an idea world (beware, I am switching of "pragmatic thinking mode" 
 for a moment), I would describe the situation would as follows:
 
 Either part can be compiled in "debug" mode or in "release" mode. Debug 
 mode in the library means that you want to debug the library code 
 itself. Release mode in the library means that you trust the library 
 code to be correct and switch off all internal checks.


I see the choice of "release" for disabling contracts as a huge mistake in
nomenclature.  For libraries, I would ship a checked and unchecked build (with
-release disabled and enabled), but none with -debug or -unittest set.  Those
are for internal testing and the user shouldn't care to turn on debug code in a
library simply because he's debugging his own app.

The idea of compiling the "in" contract into the application code is an
interesting one, but I suspect it could be tricky.  Consider an unchecked build
of the library, a checked build of the app, and now taking the address of a
library function.  Worse, what if a library routine returns the address of
another library routine?  Now the application has a reference to an unchecked
version of the function, even if the involved technical hurdles are surmounted
(multiple entry points or the like).

This seems like a nice idea but it seems too complicated.  I'd rather just ship
checked and unchecked builds of a library and leave it at that.

Jun 30 2010

Norbert Nemec <Norbert Nemec-online.de> writes:

On 30/06/10 17:45, Sean Kelly wrote:
 Norbert Nemec Wrote:

 On 28/06/10 12:59, bearophile wrote:
 Norbert Nemec:
 [...] to place code for input contract checking in the *calling* code. [...]
 Output contract checks, on the other hand should be compiled inside the
 returning routine.


 Is this a positive thing to do? Can this be done? (D must support separate
compilation, but in many situations this is not done/necessary, so maybe in
such situations it can be done). Is Eiffel doing it? if it's a good thing and
it's doable then what kind of changes does it require to the compiler?


 These are good and pragmatic questions that you ask.

 The whole issue only arises when doing separate compilation of a library
 and an application. (I use the term "application" for any code that uses
 the library.)

 In an idea world (beware, I am switching of "pragmatic thinking mode"
 for a moment), I would describe the situation would as follows:

 Either part can be compiled in "debug" mode or in "release" mode. Debug
 mode in the library means that you want to debug the library code
 itself. Release mode in the library means that you trust the library
 code to be correct and switch off all internal checks.


 I see the choice of "release" for disabling contracts as a huge mistake in
nomenclature.  For libraries, I would ship a checked and unchecked build (with
-release disabled and enabled), but none with -debug or -unittest set.  Those
are for internal testing and the user shouldn't care to turn on debug code in a
library simply because he's debugging his own app.

 The idea of compiling the "in" contract into the application code is an
interesting one, but I suspect it could be tricky.  Consider an unchecked build
of the library, a checked build of the app, and now taking the address of a
library function.  Worse, what if a library routine returns the address of
another library routine?  Now the application has a reference to an unchecked
version of the function, even if the involved technical hurdles are surmounted
(multiple entry points or the like).


That's indeed an interesting aspect: Design by Contract (DbC) and 
function pointers. I am not sure how these concepts would merge properly 
at all.

The contracts are part of the interface, so they should in fact be part 
of the function pointer type! Of course this would quickly become 
ridiculous.

A strongly object oriented language like Eiffel can in principle do 
without function pointers. Instead, one can in most cases use classes 
with virtual functions that offer very similar functionality. A class 
interface comes with all the contracts, so everything is safe and sound.

I really do not know how to deal with function pointers in the clean DbC 
paradigm. If you assign a function with input contracts to a function 
pointer, whoever uses the pointer does not know about the contracts. 
This however, breaks down the strong DbC concept and turns contracts 
into mere run time checks.

Does this mean that D should give up the goal of proper DbC? Simply do 
the pragmatic thing and pick the best pieces from DbC without worrying 
about formal completeness? I guess so...

Jun 30 2010

bearophile <bearophileHUGS lycos.com> writes:

Jay Byrd:
 This is all very confused, and is reflected in D implementing contracts all
wrong.


If you know well the ideas of DbC, and you think there are some problems in the
DbC of D2, then I suggest you to not just write what's wrong, why it is wrong
and what bad things such wrong design may cause, and what you suggest to
change, starting from the most important changes.

Maybe Walter will change nothing, but if you write just a little rant where you
say that all is wrong, probably nothing will change, and what you have written
is useless.

Bye,
bearophile

Sep 11 2010

bearophile <bearophileHUGS lycos.com> writes:

 then I suggest you to not just write what's...


Ignore that 'not', please.

Sep 11 2010

Norbert Nemec <Norbert Nemec-online.de> writes:

On 11/09/10 08:18, Jay Byrd wrote:
 Contracts do not belong to function pointers or any other
 dynamic state -- they apply to the invoker, and thus the static type.
 Isn't that obvious?


In fact, it is yet one step more complex than that: as the name itself 
suggests, contracts are "between" the caller (static type) and the 
callee (dynamic type). Theoretically, the type system has to ensure that 
both sides are able to fulfill their part of the contract.

The dynamic type must be a subtype of the static type (including 
equality). In subtyping, in-contracts may be weakened, out-contracts may 
be strengthened (in other words, a subtype may require less and promise 
more).

This is all fine, theoretically sound and easy to handle in a clean way 
for object oriented design as it is done in Eiffel.

The complication in D are function pointers and delegates (FP/DG). For a 
clean design, the type of a FP/DG would need to include contract 
information. Contracts are part of the interface and a FP/DG would have 
to include this. Obviously, this would make FP/DG syntax rather awkward. 
Furthermore, FP/DG assignments would need to be type-checked at compile 
time, so contract compatibility would have to be checked at compile time 
as well. This would be completely impossible.

I conclude that within pure OOP, contracts can have strong compile-time 
support. In-contracts should be checked by the caller, out-contracts by 
the callee and both checks could be eliminated if the compiler can 
verify at compile time that they are fulfilled. With FP/DG, this breaks 
down and I believe the best one can do is to implement contracts as 
run-time checks in the callee, just as it is done in D.

The only detail that I would wish for is a more fine-grained tuning of 
DbC contract checks in the compiler and a clearer conceptual separation 
of the concepts of assertions and contracts. However, the former is an 
implementation detail and the latter has been discussed to death before.

Sep 11 2010

bearophile <bearophileHUGS lycos.com> writes:

Norbert Nemec:

Thank you for your clear explanations of the situation.

 With FP/DG, this breaks 
 down and I believe the best one can do is to implement contracts as 
 run-time checks in the callee, just as it is done in D.


There is also a mixed strategy: to use run-time checks in the callee for FP/DG,
and something better for the other situations (FP/DG are present in D2
programs, but they aren't used everywhere).

Bye,
bearophile

Sep 12 2010

Norbert Nemec <Norbert Nemec-online.de> writes:

On 12/09/10 14:48, bearophile wrote:
 Norbert Nemec:

 Thank you for your clear explanations of the situation.

 With FP/DG, this breaks
 down and I believe the best one can do is to implement contracts as
 run-time checks in the callee, just as it is done in D.


 There is also a mixed strategy: to use run-time checks in the callee for
FP/DG, and something better for the other situations (FP/DG are present in D2
programs, but they aren't used everywhere).


Indeed - this would mean a bare, unchecked interface for each function 
and a wrapper that adds contract checks. If the calling code can verify 
the contract at compile time, it may call the unchecked version. 
Otherwise (like with FP/DG), it will call the checked wrapper.

In fact, this could be understood like a kind of type-casting: A 
function interface includes the contract as part of the type 
information. If the contract is "casted" away (by assigning the function 
to a FP that does not include a contract) the the FP points to the 
run-time checked version of the routine.

This concept may even open the road towards FP/DG that include contract 
information without getting in the way of lightweight, contract-free 
FP/DGs as we have them now.

Greetings,
Norbert

Sep 12 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Mon, 21 Jun 2010 16:30:48 +0300, Leandro Lucarella <luca llucax.com.ar>  
wrote:

 Andrei Alexandrescu, el 21 de junio a las 08:02 me escribiste:
 On 06/20/2010 11:08 PM, Leandro Lucarella wrote:
Walter Bright, el 20 de junio a las 19:32 me escribiste:
Leandro Lucarella wrote:
Why will you assume I'm so dumb that I won't use your
interface correctly?


Windows has had major legacy compatibility issues because critical
third party applications misused the APIs.

People *will* misuse your API, and you will get blamed for it. It's
unfair, but that's how it works.


Luckily I haven't used Windows for about 10 years now =)

It's really a shame that D will take the stupidity route.

PS: I don't know how windows work, but if calling the Windows API is
     like going into kernel mode, and you can mess other processes, it
     seems reasonable to do check every API call as if it were user
     input, but if you're confined to your process, is really stupid.


 Why is it stupid?


 Because you're adding unnecessary extra checks, just based on
 (Windows?) programmer's stupidity.


Walter makes a good point. If someone uses your API in the wrong way and  
relies on undocumented/undefined behavior, you'll end up having to support  
this usage pattern in future implementations of your interface if you want  
businesses and other entities who depend on that product to buy your new  
operating system.

-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 21 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Sun, 20 Jun 2010 03:04:31 +0300, Walter Bright  
<newshound1 digitalmars.com> wrote:

 Andrei Alexandrescu wrote:
 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 its parameters as user input. Therefore they need to be scrubbed with  
 hard tests, not contracts.


 I should add that any library that may be used as a dll should have its  
 interface API checked with hard tests, not contracts. This is because a  
 dll cannot control who connects to it, and therefore must regard  
 anything sent to it as unvalidated user input.


I don't see the logic in this...

Are we talking about validating user input for the sake of security, or  
debugging (catching bugs early)?

If it's for the sake of debugging, debug checks should remain in debug  
builds (that's what contracts are for?). Otherwise, you are stripping the  
programmer of the choice between higher performance or more debug checks.

If it's for the sake of security - parameter validation in DLLs is  
pointless. If you are able to load and call code from inside a DLL, you  
are already able to do everything that the DLL can. DLLs don't have any  
"setuid"-like properties. If we were talking, for example, about syscalls  
for a kernel module (functions called from userland but executed in kernel  
land), then that would be a completely different situation.

Also, I don't think that one rule can apply for everyone. For example, a  
high-performance DLL may specify in its documentation that the function  
parameters are not checked by the DLL and must be valid, otherwise  
undefined behavior will occur. (I believe some Windows APIs do not check  
some parameters and will cause access violations when called with invalid  
parameters.)

-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 19 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright  
<newshound2 digitalmars.com> wrote:

 An input to a dll is user input, and should be validated (for the sake  
 of security, and other reasons). Validating it is not debugging.


I don't understand why you're saying this. Security checks in DLL  
functions are pointless, for the reasons I already outlined:

  If it's for the sake of security - parameter validation in DLLs is  
 pointless. If you are able to load and call code from inside a DLL, you  
 are already able to do everything that the DLL can. DLLs don't have any  
 "setuid"-like properties. If we were talking, for example, about  
 syscalls for a kernel module (functions called from userland but  
 executed in kernel land), then that would be a completely different  
 situation.


 If you, for example, provide a pluggable interface to your browser app,  
 that's done using a dll, and you'd better validate anything you get  
 through that plugin interface!


Why? When your application loads a DLL, the DLL instantly gets access to  
all of your application's memory, handles, and other resources. It's  
running in the same address space and security context. You need to  
completely trust the DLL - which is why new browsers (Google Chrome and  
experimental Firefox versions) load plugins in separate processes with  
reduced privileges.

-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 20 2010

BCS <none anon.com> writes:

Hello Vladimir,

 On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright
 <newshound2 digitalmars.com> wrote:
 
 An input to a dll is user input, and should be validated (for the
 sake  of security, and other reasons). Validating it is not
 debugging.
 


 functions are pointless, for the reasons I already outlined:
 


import my.dll;

void fn()
{
    auto data = get.userUncheckedInput();
    my.dll.doSomething(data); // if doSomething dosn't check it's inputs, 
then this can cause a security flaw
}

Yes that's your dll's user's fault but adding the checks solves it even so. 
To boot, it reduces your support cost (as long as people read error message) 
and prevents the user from having to debug starting deep inside your dll.

 If it's for the sake of security - parameter validation in DLLs is
 pointless. If you are able to load and call code from inside a DLL,
 you  are already able to do everything that the DLL can. DLLs don't
 have any  "setuid"-like properties. If we were talking, for example,
 about  syscalls for a kernel module (functions called from userland
 but  executed in kernel land), then that would be a completely
 different  situation.
 


 app,  that's done using a dll, and you'd better validate anything you
 get  through that plugin interface!
 


 to  all of your application's memory, handles, and other resources.
 It's  running in the same address space and security context. You need
 to  completely trust the DLL - which is why new browsers (Google
 Chrome and  experimental Firefox versions) load plugins in separate
 processes with  reduced privileges.


And you can bet that every byte of data shipped back and forth via IPC is 
validated more than an air traveler at a TSA checkpoint.

As for the case where the dll is local, never attribute to malice that which 
can be adequately explained by stupidity. Unless you have source, you can't 
assume that the data coming out doesn't conation unvalidated user input and 
you should always assume that someone malicious will get ahold of that sooner 
or later.  
 
-- 
... <IXOYE><

Jun 20 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Mon, 21 Jun 2010 03:02:53 +0300, BCS <none anon.com> wrote:

 import my.dll;

 void fn()
 {
     auto data = get.userUncheckedInput();
     my.dll.doSomething(data); // if doSomething dosn't check it's  
 inputs, then this can cause a security flaw
 }

 Yes that's your dll's user's fault but adding the checks solves it even  
 so. To boot, it reduces your support cost (as long as people read error  
 message) and prevents the user from having to debug starting deep inside  
 your dll.


A well-designed application needs to validate unsafe user input exactly  
once (assuming the process of validation is the same).
DLL interfaces must specify whether the input/output can be considered  
safe or not.
Not doing so results in either security holes or redundant code.

 And you can bet that every byte of data shipped back and forth via IPC  
 is validated more than an air traveler at a TSA checkpoint.


Obviously, no argument here.

 As for the case where the dll is local, never attribute to malice that  
 which can be adequately explained by stupidity. Unless you have source,  
 you can't assume that the data coming out doesn't conation unvalidated  
 user input and you should always assume that someone malicious will get  
 ahold of that sooner or later.


Unless you have the source, you can't assume that simply loading the DLL  
will not create a security hole.

Trusting the DLL but not trusting the data it gives you is a plausible  
case. As I said before, this simply needs to be well-defined, and  
validation doesn't have to happen exactly at the DLL boundary.


-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 20 2010

BCS <none anon.com> writes:

Hello Vladimir,

 On Mon, 21 Jun 2010 03:02:53 +0300, BCS <none anon.com> wrote:
 
 import my.dll;
 
 void fn()
 {
 auto data = get.userUncheckedInput();
 my.dll.doSomething(data); // if doSomething dosn't check it's
 inputs, then this can cause a security flaw
 }
 Yes that's your dll's user's fault but adding the checks solves it
 even  so. To boot, it reduces your support cost (as long as people
 read error  message) and prevents the user from having to debug
 starting deep inside  your dll.
 


 exactly
 once (assuming the process of validation is the same).
 DLL interfaces must specify whether the input/output can be considered
 safe or not.
 Not doing so results in either security holes or redundant code.


If I didn't write the DLL I'm calling, I'll assume it doesn't check stuff. 
If I didn't write the code calling my DLL, I'll assume it doesn't check stuff. 
Why should I assume that the documentation is right or that people will even 
read my documentation? Unless you can show me that this causes a perf problem, 
the benefit just isn't worth the cost.

 And you can bet that every byte of data shipped back and forth via
 IPC  is validated more than an air traveler at a TSA checkpoint.
 


 
 As for the case where the dll is local, never attribute to malice
 that  which can be adequately explained by stupidity. Unless you have
 source,  you can't assume that the data coming out doesn't conation
 unvalidated  user input and you should always assume that someone
 malicious will get  ahold of that sooner or later.
 


 DLL  will not create a security hole.


OK, but just because there might be a risk in loading a DLL is no reason 
to not address other risk that you can deal with.

 
 Trusting the DLL but not trusting the data it gives you is a plausible
 case. As I said before, this simply needs to be well-defined, and
 validation doesn't have to happen exactly at the DLL boundary.


Good point, I might not check it exactly at the call site, but *I* will will 
check it because I will assume that any checks on the other side of a DLL 
interface are flawed, missing, broken or just flat wrong.

-- 
... <IXOYE><

Jun 20 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Mon, 21 Jun 2010 04:53:46 +0300, BCS <none anon.com> wrote:

 If I didn't write the DLL I'm calling, I'll assume it doesn't check  
 stuff. If I didn't write the code calling my DLL, I'll assume it doesn't  
 check stuff. Why should I assume that the documentation is right or that  
 people will even read my documentation? Unless you can show me that this  
 causes a perf problem, the benefit just isn't worth the cost.


If you can't trust the DLL to perform correct user data validation, you  
can't trust it AT ALL! For all you know it can have a buffer overflow  
vulnerability. Re-validating any data you get from it may save you from  
one type of bug, but it doesn't improve security by much overall.

Regarding performance: what is not a "performance problem" in any one  
single place can make a considerable difference when you sum up all the  
redundant checks in your entire codebase.

A practical example from the industry:

For Microsoft partners, Windows is available in a "free" (or retail) build  
and a "checked" build [1].
Since driver code runs in kernel space, drivers can crash the entire  
system anyway - for this reason, there are few checks or kernel mode APIs.  
(That's why sometimes when debugging BSoDs, crashes will happen in a  
completely unrelated kernel module.)
However, if you need to debug your driver, you run it on the checked  
version of Windows, which additionally to having lots of debug checks is  
also built without most compiler optimizations.

   [1]: http://msdn.microsoft.com/en-us/library/ff543450(VS.85).aspx

-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 21 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Mon, 21 Jun 2010 03:40:48 +0300, Walter Bright  
<newshound2 digitalmars.com> wrote:

 Vladimir Panteleev wrote:
 On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright  
 <newshound2 digitalmars.com> wrote:

 An input to a dll is user input, and should be validated (for the sake  
 of security, and other reasons). Validating it is not debugging.


 functions are pointless, for the reasons I already outlined:


 It's true that whenever user code is executed, that code can do  
 anything. Hello, ActiveX. But I still think it's sound practice to treat  
 any data received from another program as untrusted, and validate it.  
 Security, like I said, is only one reason. Another is to prevent bugs in  
 external code from trashing your process.


Yes, but this is a completely different kind of trust (incompetence  
instead of intentional malice) :)

I was simply arguing the technical point of pointlessness of verifying  
data from DLLs specifically for security reasons (buffer overflows, code  
injection etc.).

Other than that, this is the usual performance vs. robustness dilemma  
(though my personal opinion is that an ideal language/platform/etc. should  
allow programmers to take all the responsibility for maximum performance).

-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 20 2010

"Rory McGuire" <rmcguire neonova.co.za> writes:

On Mon, 21 Jun 2010 06:08:01 +0200, Leandro Lucarella <luca llucax.com.ar>  
wrote:

 Walter Bright, el 20 de junio a las 19:32 me escribiste:
 Leandro Lucarella wrote:
Why will you assume I'm so dumb that I won't use your
interface correctly?


 Windows has had major legacy compatibility issues because critical
 third party applications misused the APIs.

 People *will* misuse your API, and you will get blamed for it. It's
 unfair, but that's how it works.


 Luckily I haven't used Windows for about 10 years now =)

 It's really a shame that D will take the stupidity route.

 PS: I don't know how windows work, but if calling the Windows API is
     like going into kernel mode, and you can mess other processes, it
     seems reasonable to do check every API call as if it were user
     input, but if you're confined to your process, is really stupid.


I think perhaps you mis-understood, it is mostly not stupidity that causes  
people to use
undocumented "features" of an API but rather, it is people being overly  
"clever".

Jun 21 2010

Leandro Lucarella <luca llucax.com.ar> writes:

Andrei Alexandrescu, el 21 de junio a las 08:02 me escribiste:
 On 06/20/2010 11:08 PM, Leandro Lucarella wrote:
Walter Bright, el 20 de junio a las 19:32 me escribiste:
Leandro Lucarella wrote:
Why will you assume I'm so dumb that I won't use your
interface correctly?


Windows has had major legacy compatibility issues because critical
third party applications misused the APIs.

People *will* misuse your API, and you will get blamed for it. It's
unfair, but that's how it works.


Luckily I haven't used Windows for about 10 years now =)

It's really a shame that D will take the stupidity route.

PS: I don't know how windows work, but if calling the Windows API is
     like going into kernel mode, and you can mess other processes, it
     seems reasonable to do check every API call as if it were user
     input, but if you're confined to your process, is really stupid.


 Why is it stupid?


Because you're adding unnecessary extra checks, just based on
(Windows?) programmer's stupidity.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
JUGAR COMPULSIVAMENTE ES PERJUDICIAL PARA LA SALUD.
	-- Casino de Mar del Plata

Jun 21 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Mon, 21 Jun 2010 21:15:12 +0300, Adrian Matoga <epi atari8.info> wrote:

 It was 15 years ago, at the times of 3.x and 95, when Windows behaved  
 like that.


More like 10, Windows Millennium was the last 9x-based Windows operating  
system without strong process isolation.

 The problem applies not only to Windows, but any API you would imagine.
 A common situation is when you need to do your job quickly using only  
 some part of a library which otherwise you aren't going to study  
 thoroughly, or you want only a proof-of-concept. And if your attempting  
 to use something new to you, you do make mistakes, no matter how you are  
 convinced yo do not.
 If the API is defined not by documentation (which is often a tissue of  
 lies, and hardly ever it's unambiguous), but by means of input checking,  
 you have benefits in two fields: 1) developers of library had to think  
 what they wanted to do, so library probably works, and it's less  
 probable that its new versions will break the compatiblity, and 2) users  
 of the library will be warned quickly, saving their time.

 It's not about messing other processes. It's about saving your time,  
 otherwise consumed by effects of common mistakes, misunderstanding the  
 documentation, or working in a hurry. And your time costs much more than  
 the time of bazillion argument checks.


For this particular situation, contracts would be just fine though :)

-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 21 2010

Norbert Nemec <Norbert Nemec-online.de> writes:

On 19/06/10 22:46, Andrei Alexandrescu wrote:
 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 Walter and I discussed this and concluded that Phobos should handle its
 parameters as user input. Therefore they need to be scrubbed with hard
 tests, not contracts.


IMHO, this is plain wrong!

By this kind of decision, you are putting the library user under 
tutelage. The D language is explicitly designed to allow the user to 
take off the safety belt, but to do so at their own responsibility. I 
typically get very annoyed out of principle if I explicitly and 
consciously switch *off* a safety feature and someone decides that they 
do not trust that I am mature enough for that decision and leave part of 
the safety checks in place.

The interface between parts of a program is exactly what contracts are 
designed for. Calling a function with incorrect arguments is a bug in 
the calling code. It should be caught at the interface by a contract 
violation. If the library user trusts their program enough to switch off 
contract checking, the library designer should not worry about 
double-checking for incorrect function calls.

Contracts are part of the public interface, so whatever is specified 
there is automatically documented for the world to see. Replacing 
contracts by "enforce" statements inside the library functions means 
that you have to document in prose what kind of input arguments are allowed.

There are exceptional cases when exceptions may be thrown for incorrect 
arguments: typically, whenever testing for input correctness would be 
too costly, e.g. for badly conditioned matrices in linear algebra code. 
Here, the problem typically shows up during the calculation when it is 
too late to issue a contract violation.

Otherwise, exceptions in library code should only happen in well-defined 
cases for run-time conditions (like I/O errors).

Sorry about by blunt words, but I feel that behind this issue there 
still is a rather fundamental misunderstanding of the concepts of 
contract programming.

Jun 28 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 06/28/2010 03:15 AM, Norbert Nemec wrote:
 On 19/06/10 22:46, Andrei Alexandrescu wrote:
 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 Walter and I discussed this and concluded that Phobos should handle its
 parameters as user input. Therefore they need to be scrubbed with hard
 tests, not contracts.


 IMHO, this is plain wrong!

 By this kind of decision, you are putting the library user under
 tutelage. The D language is explicitly designed to allow the user to
 take off the safety belt, but to do so at their own responsibility.


C APIs also check their arguments.

Andrei

Jun 28 2010

bearophile <bearophileHUGS lycos.com> writes:

Andrei Alexandrescu:
 C APIs also check their arguments.


Try again, C doesn't have DbC :-) Norbert Nemec says some good things.

Bye,
bearophile

Jun 28 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

bearophile wrote:
 Andrei Alexandrescu:
 C APIs also check their arguments.


 Try again, C doesn't have DbC :-)


What I meant to say was that even the standard library of a language 
famous for its to-the-metal performance still checks parameters 
compulsively whenever it can. Search e.g. this page:

http://google.com/codesearch/p?hl=en#XAzRy8oK4zA/libc/stdio/fseek.c&q=fseek&sa=N&cd=2&ct=rc

for "whence".

 Norbert Nemec says some good things.


I think it's an area where reasonable people may disagree.


Andrei

Jun 28 2010

Norbert Nemec <Norbert Nemec-online.de> writes:

On 28/06/10 19:30, Andrei Alexandrescu wrote:
 bearophile wrote:
 Andrei Alexandrescu:
 C APIs also check their arguments.


 Try again, C doesn't have DbC :-)


 What I meant to say was that even the standard library of a language
 famous for its to-the-metal performance still checks parameters
 compulsively whenever it can. [...]


Indeed, checking input arguments is essential. DbC simply means 
formalizing what has been done in any good libary for ages.

My only intention was to make clear that checking input arguments is 
exactly what contracts are designed for. If the Phobos designers are 
worried that their input checks might be deactivated prematurely, we 
should fix the policy for (de/)activating the input contract checks 
rather than avoiding the use of input contracts they way they are 
intended to be used.

"enforce" simply is not the right tool for this purpose: When a contract 
or and assertion violation is found, it is clear that there is a bug in 
the code. Furthermore, it is even clear which portion of the code is 
responsible for the bug. An assertion violation inside a library is a 
bug in this library. This bug may simply be a missing input contract, 
but this still is a bug in the library.

An "enforce" violation in the library may be anything. The user has to 
dig into the library code to find out whether it is a library bug or a 
piece of incorrect input.

Raising a well-defined exception gives enough information about the 
problem. An anonymous "enforce" on the other hand is a quick-and-dirty 
solution that does not help the developer very much to identify the real 
problem.


 Norbert Nemec says some good things.


 I think it's an area where reasonable people may disagree.


Thanks bearophile, thanks Andrei as well -- I really appreciate this 
open exchange of ideas. Feel free to shoot back as directly as I 
attacked... :-)

Jun 30 2010

Michel Fortin <michel.fortin michelf.com> writes:

On 2010-06-28 07:17:53 -0400, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 On 06/28/2010 03:15 AM, Norbert Nemec wrote:
 On 19/06/10 22:46, Andrei Alexandrescu wrote:
 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 Walter and I discussed this and concluded that Phobos should handle its
 parameters as user input. Therefore they need to be scrubbed with hard
 tests, not contracts.


 IMHO, this is plain wrong!
 
 By this kind of decision, you are putting the library user under
 tutelage. The D language is explicitly designed to allow the user to
 take off the safety belt, but to do so at their own responsibility.


 C APIs also check their arguments.


With C you don't have the option to turn the checks on or off. It's 
generally better to have them when you don't need them than not have 
them when you need them. With D, you can turn them on or off on a whim.

If the 'in' contract was enforced at the call site instead of inside 
the function, it'd be up to the one using a function to decide whether 
to check contracts or not. That's not an option in C, but it could work 
like that in D...

I agree though that with the way contracts are currently implemented 
this doesn't work very well. You have to recompile the library with 
contracts on, which in turn forces all the internal contracts inside 
the library to be evaluated. All this because you're trying to validate 
inputs you give to that library? Doesn't make sense. In that context I 
agree that checking explicitly at the library boundaries might be a 
more viable option (like in C).

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/

Jun 28 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

Michel Fortin wrote:
 On 2010-06-28 07:17:53 -0400, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:
 
 On 06/28/2010 03:15 AM, Norbert Nemec wrote:
 On 19/06/10 22:46, Andrei Alexandrescu wrote:
 On 06/19/2010 03:55 PM, bearophile wrote:
 Inside Phobos2 I have counted about 160 usages of the "body" keyword.
 I think contract programming can be used more often inside Phobos2
 (and maybe some usages of enforce() can be turned into contract
 programming because they are more similar to program sanity checks).


 Walter and I discussed this and concluded that Phobos should handle its
 parameters as user input. Therefore they need to be scrubbed with hard
 tests, not contracts.


 IMHO, this is plain wrong!

 By this kind of decision, you are putting the library user under
 tutelage. The D language is explicitly designed to allow the user to
 take off the safety belt, but to do so at their own responsibility.


 C APIs also check their arguments.


 With C you don't have the option to turn the checks on or off.


#define NDEBUG


Andrei

Jun 28 2010

Sean Kelly <sean invisibleduck.org> writes:

Andrei Alexandrescu Wrote:
 
 C APIs also check their arguments.


Not the standard C library, as far as I know.  Of course, it's also gotten a
lot of flak for this.

Jun 28 2010

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

Sean Kelly wrote:
 Andrei Alexandrescu Wrote:
 C APIs also check their arguments.


 Not the standard C library, as far as I know.  Of course, it's also gotten a
lot of flak for this.


Nonono. They check whenever they can. Oftentimes they're unable to check.

Example: fseek checks its whence parameter (mentioned in my previous 
post) but itoa cannot check that the target is a valid memory buffer of 
the appropriate length.


Andrei

Jun 28 2010

"Simen kjaeraas" <simen.kjaras gmail.com> writes:

Norbert Nemec <Norbert nemec-online.de> wrote:

 A library designer should trust the library user to act responsibly.
 After all - if the application breaks it is the application designer
 who has to answer for it.


And if the application designer finds that his design breaks due to
a change in the library, he will blame the library designers. If it
is used in a big, well-known and much-used application, the library
designer might have no choice but to continue supporting broken
code.

-- 
Simen

Jun 28 2010

"Danny Wilson" <danny decube.net> writes:

Op Mon, 28 Jun 2010 10:33:24 +0200 schreef Norbert Nemec  
<Norbert nemec-online.de>:

 Conceptually, the ultimate solution would certainly be to place code for  
 input contract checking in the *calling* code. After all, this checking  
 code serves to debug the calling code, so it should be left to the  
 caller to decide whether checking is necessary.



 This approach would also allow the compiler to optimize out some checks  
 when their correctness can be tested at compile time.

 Output contract checks, on the other hand should be compiled inside the  
 returning routine.

 After all, it is all a matter of trust. A language designer should trust  
 the language user to know what he is doing. A library designer should  
 trust the library user to act responsibly. After all - if the  
 application breaks it is the application designer who has to answer for  
 it.




Alot. :-)

Jun 28 2010

Jay Byrd <JayByrd rebels.com> writes:

On Wed, 30 Jun 2010 20:03:07 +0100, Norbert Nemec wrote:

 On 30/06/10 17:45, Sean Kelly wrote:
 Norbert Nemec Wrote:

 On 28/06/10 12:59, bearophile wrote:
 Norbert Nemec:
 [...] to place code for input contract checking in the *calling*
 code. [...] Output contract checks, on the other hand should be
 compiled inside the returning routine.


 Is this a positive thing to do? Can this be done? (D must support
 separate compilation, but in many situations this is not
 done/necessary, so maybe in such situations it can be done). Is
 Eiffel doing it? if it's a good thing and it's doable then what kind
 of changes does it require to the compiler?


 These are good and pragmatic questions that you ask.

 The whole issue only arises when doing separate compilation of a
 library and an application. (I use the term "application" for any code
 that uses the library.)

 In an idea world (beware, I am switching of "pragmatic thinking mode"
 for a moment), I would describe the situation would as follows:

 Either part can be compiled in "debug" mode or in "release" mode.
 Debug mode in the library means that you want to debug the library
 code itself. Release mode in the library means that you trust the
 library code to be correct and switch off all internal checks.


 I see the choice of "release" for disabling contracts as a huge mistake
 in nomenclature.  For libraries, I would ship a checked and unchecked
 build (with -release disabled and enabled), but none with -debug or
 -unittest set.  Those are for internal testing and the user shouldn't
 care to turn on debug code in a library simply because he's debugging
 his own app.

 The idea of compiling the "in" contract into the application code is an
 interesting one, but I suspect it could be tricky.  Consider an
 unchecked build of the library, a checked build of the app, and now
 taking the address of a library function.  Worse, what if a library
 routine returns the address of another library routine?  Now the
 application has a reference to an unchecked version of the function,
 even if the involved technical hurdles are surmounted (multiple entry
 points or the like).


 That's indeed an interesting aspect: Design by Contract (DbC) and
 function pointers. I am not sure how these concepts would merge properly
 at all.
 
 The contracts are part of the interface, so they should in fact be part
 of the function pointer type! Of course this would quickly become
 ridiculous.
 
 A strongly object oriented language like Eiffel can in principle do
 without function pointers. Instead, one can in most cases use classes
 with virtual functions that offer very similar functionality. A class
 interface comes with all the contracts, so everything is safe and sound.
 
 I really do not know how to deal with function pointers in the clean DbC
 paradigm. If you assign a function with input contracts to a function
 pointer, whoever uses the pointer does not know about the contracts.
 This however, breaks down the strong DbC concept and turns contracts
 into mere run time checks.
 
 Does this mean that D should give up the goal of proper DbC? Simply do
 the pragmatic thing and pick the best pieces from DbC without worrying
 about formal completeness? I guess so...


This is all very confused, and is reflected in D implementing contracts 
all wrong. Contracts do not belong to function pointers or any other 
dynamic state -- they apply to the invoker, and thus the static type. 
Isn't that obvious? If I have Foo f = getSomeFoo(); result = f.method
(args), the args must satisfy the contract for Foo.method(), regardless 
of what method() in the object returned by getSomeFoo() is willing to 
accept (it must, of course, not require more than Foo.method() does; TDPL 
at least gets that right). And the guarantees on result must be those 
promised by Foo.method(), not some much weaker promise given by method() 
in its base class (and there is no need to check stronger guarantees made 
by method() the actual derived object, since the caller didn't ask for 
them).

The D implementation works much too hard to get the wrong result, both in 
complexities of the compiler and in executing a bunch of irrelevant code 
that is ignored if it fails in preconditions or succeeds in 
postconditions, and after all that it fails to enforce requirements it 
should and to guarantee promises that it should.

-- JB

Sep 10 2010

retard <re tard.com.invalid> writes:

Sat, 11 Sep 2010 07:16:56 -0400, bearophile wrote:

 Jay Byrd:
 This is all very confused, and is reflected in D implementing contracts
 all wrong.


 If you know well the ideas of DbC, and you think there are some problems
 in the DbC of D2, then I suggest you to not just write what's wrong, why
 it is wrong and what bad things such wrong design may cause, and what
 you suggest to change, starting from the most important changes.
 
 Maybe Walter will change nothing, but if you write just a little rant
 where you say that all is wrong, probably nothing will change, and what
 you have written is useless.
 
 Bye,
 bearophile


People are doing it wrong. They shouldn't come and rant here. They should 
write compiler patches instead. All discussion is bad, real code matters.

Sep 11 2010

Leandro Lucarella <luca llucax.com.ar> writes:

Walter Bright, el 20 de junio a las 17:40 me escribiste:
 Vladimir Panteleev wrote:
On Mon, 21 Jun 2010 00:17:28 +0300, Walter Bright
<newshound2 digitalmars.com> wrote:

An input to a dll is user input, and should be validated (for
the sake of security, and other reasons). Validating it is not
debugging.


I don't understand why you're saying this. Security checks in DLL
functions are pointless, for the reasons I already outlined:


 It's true that whenever user code is executed, that code can do
 anything. Hello, ActiveX. But I still think it's sound practice to
 treat any data received from another program as untrusted, and
 validate it. Security, like I said, is only one reason. Another is
 to prevent bugs in external code from trashing your process.


How can you prevent that? If you pass incorrect data to a DLL, then the
bug is *yours*. If the DLL has a bug, it will explode anyways. You are
just trying to catch programs bugs in the DLL, which seems overly
patronizing to me. Why will you assume I'm so dumb that I won't use your
interface correctly?

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Karma police
arrest this girl,
her Hitler hairdo
is making me feel ill
and we have crashed her party.

Jun 20 2010

BCS <none anon.com> writes:

Hello Leandro,

 If the DLL has a bug, it will explode anyways.


A DLL can work just fine (a.k.a. not explode) and still return garbage as 
long as it never depends on not seeing the kind of garbage it's producing. 
Say for instance it's written in D and returns a string with a missing \0 
as a char*/length, the DLL works just fine but a printf blows up.

 You
 are just trying to catch programs bugs in the DLL,


Exactly.

 which seems overly
 patronizing to me.


Um, it doesn't to me.

 Why will you assume I'm so dumb that I won't use your interface correctly?


First because some people are. And second, because it trivially easy to respond 
to support calls that start with "Your DLL is throwing a
YouAreNotUsingThisDLLCorrectlyRTFM 
Exception" <joke/>.

-- 
... <IXOYE><

Jun 20 2010

"Vladimir Panteleev" <vladimir thecybershadow.net> writes:

On Mon, 21 Jun 2010 07:42:34 +0300, BCS <none anon.com> wrote:

 Why will you assume I'm so dumb that I won't use your interface  
 correctly?


 First because some people are. And second, because it trivially easy to  
 respond to support calls that start with "Your DLL is throwing a  
 YouAreNotUsingThisDLLCorrectlyRTFM Exception" <joke/>.


I think that for such situations you should ship a debug and release  
version of the DLL.
This way you don't sacrifice performance when the user doesn't want to be  
held by the hand.

-- Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net

Jun 21 2010

BCS <none anon.com> writes:

Hello Vladimir,

 On Mon, 21 Jun 2010 07:42:34 +0300, BCS <none anon.com> wrote:
 
 Why will you assume I'm so dumb that I won't use your interface
 correctly?
 


 to  respond to support calls that start with "Your DLL is throwing a
 YouAreNotUsingThisDLLCorrectlyRTFM Exception" <joke/>.
 


 version of the DLL.
 This way you don't sacrifice performance when the user doesn't want to
 be
 held by the hand.


Until you can show me a perf problem, I don't see any point in doing that. 
(OTOH, deep structure validation, or anything else slower than O(1), is another 
thing all together)

-- 
... <IXOYE><

Jun 21 2010

Lutger <lutger.blijdestijn gmail.com> writes:

Simen kjaeraas wrote:

 BCS <none anon.com> wrote:
 I think that for such situations you should ship a debug and release
 version of the DLL.
 This way you don't sacrifice performance when the user doesn't want to
 be
 held by the hand.


 Until you can show me a perf problem, I don't see any point in doing
 that. (OTOH, deep structure validation, or anything else slower than
 O(1), is another thing all together)


 Also, if you do have two different versions, I'll bet you ready money
 someone will program only the release version, because "that's the
 version the users will have", "the debug version is slow" or whatever
 other inane excuse their minds are capable of coming up with.
 


Naturally, debug is for debugging, not shipping. Instead one could make three 
versions:
- debug  
- release
- unsafe

Or rather let the user compile unsafe themselves if you can distribute the 
source code. 

I am sure most inane users (like me) will choose release.

Jun 22 2010

"Simen kjaeraas" <simen.kjaras gmail.com> writes:

BCS <none anon.com> wrote:
 I think that for such situations you should ship a debug and release
 version of the DLL.
 This way you don't sacrifice performance when the user doesn't want to
 be
 held by the hand.


 Until you can show me a perf problem, I don't see any point in doing  
 that. (OTOH, deep structure validation, or anything else slower than  
 O(1), is another thing all together)


Also, if you do have two different versions, I'll bet you ready money
someone will program only the release version, because "that's the
version the users will have", "the debug version is slow" or whatever
other inane excuse their minds are capable of coming up with.

-- 
Simen

Jun 22 2010

Sean Kelly <sean invisibleduck.org> writes:

"Simen kjaeraas" <simen.kjaras gmail.com> wrote:
 BCS <none anon.com> wrote:
 I think that for such situations you should ship a debug and release
 version of the DLL.
 This way you don't sacrifice performance when the user doesn't want
 to
 be
 held by the hand.


 Until you can show me a perf problem, I don't see any point in doing 
 that. (OTOH, deep structure validation, or anything else slower
 than  > O(1), is another thing all together)




 Also, if you do have two different versions, I'll bet you ready money
 someone will program only the release version, because "that's the
 version the users will have", "the debug version is slow" or whatever
 other inane excuse their minds are capable of coming up with.


What I've done with druntime is build checked and unchecked versions.  I
don't think it makes sense to ship a debug version of a library because
that's for debugging the library, not user code. I'll admit that I like
having debug symbols in place though, just not the debug tests
themselves. It's occasionally nice to not have a trace vanish just
because it passes through library code.

Jun 22 2010

Leandro Lucarella <luca llucax.com.ar> writes:

Walter Bright, el 20 de junio a las 19:32 me escribiste:
 Leandro Lucarella wrote:
Why will you assume I'm so dumb that I won't use your
interface correctly?


 Windows has had major legacy compatibility issues because critical
 third party applications misused the APIs.
 
 People *will* misuse your API, and you will get blamed for it. It's
 unfair, but that's how it works.


Luckily I haven't used Windows for about 10 years now =)

It's really a shame that D will take the stupidity route.

PS: I don't know how windows work, but if calling the Windows API is
    like going into kernel mode, and you can mess other processes, it
    seems reasonable to do check every API call as if it were user
    input, but if you're confined to your process, is really stupid.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Vaporeso sostenía a rajacincha la teoría del No-Water, la cual le
pertenecía y versaba lo siguiente: "Para darle la otra mejilla al fuego,
éste debe ser apagado con alpargatas apenas húmedas".

Jun 20 2010

D Programming

C/C++ Programming

Other

digitalmars.D - Re: enforce()?