www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Coverity tool

reply bearophile <bearophileHUGS lycos.com> writes:
Rarely I have found a so fun description of a software, how they can (barely?)
make it a commercial product, the problems they find. The program is a bug
finding tool, for C, C++, Java, C#:
http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext

I think it's an interesting read for people that are creating a new language :-)

Bye,
bearophile
Feb 08 2010
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:
 Rarely I have found a so fun description of a software, how they can
 (barely?) make it a commercial product, the problems they find. The
 program is a bug finding tool, for C, C++, Java, C#: 
 http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext
 
 
 I think it's an interesting read for people that are creating a new
 language :-)

Coverity's business model seems to be based on being very vague about what their software actually does, and being even more evasive about what they charge to use it. From what I can infer from their various statements, the primary thing it does is pair functions that must be paired, like malloc/free, lock/unlock, fopen/fclose, etc. This is ably handled in D using the scope guard statement. It does some other things like very limited array bounds checking, also already handled by D which does full array bounds checking.
Feb 09 2010
next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Walter Bright wrote:
 Coverity's business model seems to be based on being very vague about 
 what their software actually does,

Found some stuff on the Cert web site: https://www.securecoding.cert.org/confluence/display/seccode/Coverity+Prevent
Feb 09 2010
prev sibling next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:
  From what I can infer from their various statements, the primary thing 
 it does is pair functions that must be paired, like malloc/free, 
 lock/unlock, fopen/fclose, etc.

I have never used their program, and probably I will never buy it, but I have seen it used two times on the whole source code of CPython, and it has found a good amount of bugs (that devs have fixed in some weeks) despite such C sources are sometimes quite good (see for example: http://svn.python.org/view/python/trunk/Modules/_collections odule.c?view=markup ) compared to the messy C sources of Perl or Ruby. So it's an useful tool. Using D2 compiled with DMD is not an option for CPython devs, they can't even use C99. Bye, bearophile
Feb 09 2010
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:
 Walter Bright:
 From what I can infer from their various statements, the primary
 thing it does is pair functions that must be paired, like
 malloc/free, lock/unlock, fopen/fclose, etc.

I have never used their program, and probably I will never buy it, but I have seen it used two times on the whole source code of CPython, and it has found a good amount of bugs (that devs have fixed in some weeks) despite such C sources are sometimes quite good (see for example: http://svn.python.org/view/python/trunk/Modules/_collectionsmodule.c?view=markup ) compared to the messy C sources of Perl or Ruby. So it's an useful tool. Using D2 compiled with DMD is not an option for CPython devs, they can't even use C99.

I'm not disagreeing with you, I just think that Coverity doesn't have much use for D code, because what it checks for is already covered by the language.
Feb 09 2010
next sibling parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 02/09/2010 02:18 PM, Walter Bright wrote:
 bearophile wrote:
 Walter Bright:
 From what I can infer from their various statements, the primary
 thing it does is pair functions that must be paired, like
 malloc/free, lock/unlock, fopen/fclose, etc.

I have never used their program, and probably I will never buy it, but I have seen it used two times on the whole source code of CPython, and it has found a good amount of bugs (that devs have fixed in some weeks) despite such C sources are sometimes quite good (see for example: http://svn.python.org/view/python/trunk/Modules/_collectionsmodule.c?view=markup ) compared to the messy C sources of Perl or Ruby. So it's an useful tool. Using D2 compiled with DMD is not an option for CPython devs, they can't even use C99.

I'm not disagreeing with you, I just think that Coverity doesn't have much use for D code, because what it checks for is already covered by the language.

I think a large part of the article proves your point: all those checks belong in the compiler, and not in a third party tool. Still, there are lots of trivial or nontrivial or fuzzy or advanced analyses that DMD doesn't perform. A dumb example would be version(1){ pragma(msg, "1"); }else version(2){ pragma(msg, "2"); }
Feb 09 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

I just think that Coverity doesn't have much use for D code, because what it
checks for is already covered by the language.<

I have taken a look at this report about bugs found by this tool in open source code: http://scan.coverity.com/report/Coverity_White_Paper-Scan_Open_Source_Report_2009.pdf The most common bugs found there by Coverity (I think they are mostly in C code): 27.8% NULL Pointer Deference 23.3% Resource Leak 9.7% Unintentional Ignored Expressions 8.1% Use Before Test (NULL) 5.9% Use After Free 5.8% Buffer Overflow (statically allocated) 5.3% Unsafe Use of Returned NULL 8.4% Uninitialized Values Read 3.9% Unsafe Use of Returned Negative 1.1% Type and Allocation Size Mismatch 0.2% Buffer Overflow (dynamically allocated) 0.2% Use Before Test (negative) Key: - Unintentional Ignored Expressions: indicates that part of the source code is not reachable on any execution path. One example involves if statements and return statements. Consider when a programmer makes a mistake in the if() statementís test value, and the code block for the statement returns from the current function. If the test value is always true, then all the code after the if() block can never be reached. - Unsafe Use of Returned values: results of a function are not tested if they are NULL or if they are a error value (usually negative). - Use Before Test: similar to Unsafe Use of Returned values. When programmers test the value of a variable, this implies that they believe it is possible for the variable to have the undesired value at the point of the test. If a code path exists where the variable is used in an unsafe way before this is tested, then the test does not accomplish its objective. Regarding D code: - Use After Free: probably less common in D code, because dellocations are done by the GC. - Buffer Overflows: less common in D code because of array bound tests at runtime, and because some string operations/functions are safer. - Type and Allocation Size Mismatch: malloc/calloc are not common in D code, and the usage of the new statement avoids many bugs of this class. - Unsafe Use of Returned error values: hopefully in all the code (but where high-performance code is necessary) D programmers use exceptions to denote error situations, avoiding some of this class of bugs. - Resource Leak: in theory the GC can help a lot here. In practice the D GC is conservative, so it leaks by design. - Uninitialized Values Read: D "solves" this problem initializing by default all variables. In practice probably some default-initialized values can cause troubles where a default initialization was not the right thing. - NULL Pointer Deference: this probably happens in D code, by default class references can be null. You can probably find the following bugs in D code: - Use Before Test: here the bugs are in the code paths of the try-except. - Unsafe Use of Returned NULL: the D2 type system doesn't help here. - Unintentional Ignored Expressions D language surely introduces new classes of bugs absent in C code. D2 has several complex and hairy features, that can lead to bugs. ------------ Ellery Newcomer:
I think a large part of the article proves your point: all those checks belong
in the compiler, and not in a third party tool.<

In theory you are right, but in practice compiler writers have already their hands full, so I have nothing against the idea of a lint tool that does more tests compared to the compiler. The new Clang/Clang++ compiler (front-end based on LLVM. Few days ago Clang++ has compiled all llvm itself) is able to perform a good amount of (simple but useful) static tests on the code, and it gives precise error messages. Being the LLVM project very modular, with some luck a future LDC-like compiler for D2 will find a way to re-use some of the analysis modules used by Clang on D2 code, so maybe it can list unused variables, spot some cases of null reference deferences, etc. Bye, bearophile
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:
 Regarding D code:
 
 - Use After Free: probably less common in D code, because
 dellocations are done by the GC.
 - Buffer Overflows: less common in D
 code because of array bound tests at runtime, and because some string
 operations/functions are safer.

safe guarantees memory safety, so these are non-issues in D.
 - Type and Allocation Size Mismatch:
 malloc/calloc are not common in D code, and the usage of the new
 statement avoids many bugs of this class.

Using safe makes these non-issues.
 - Unsafe Use of Returned
 error values: hopefully in all the code (but where high-performance
 code is necessary) D programmers use exceptions to denote error
 situations, avoiding some of this class of bugs.

As you say, exceptions solve these problems.
 - Resource Leak: in
 theory the GC can help a lot here. In practice the D GC is
 conservative, so it leaks by design.

It is a mistake to rely on the GC to free resources other than memory. Properly, either RAII or scope guard statements should be used, as these do not leak.
 - Uninitialized Values Read: D
 "solves" this problem initializing by default all variables. In
 practice probably some default-initialized values can cause troubles
 where a default initialization was not the right thing.

No analysis tool can help you if you initialize things to the wrong value. That's very, very different from initializing things to GARBAGE and then using those garbage values.
 - NULL Pointer Deference: this probably happens in D code, by default class
 references can be null.

We've certainly had long threads about this one. I'll just state that turning on the optimizer will detect many cases of dereferencing null pointers.
 You can probably find the following bugs in D code:

 - Use Before Test: here the bugs are in the code paths of the try-except.

Not sure what this means.
 - Unsafe Use of Returned NULL: the D2 type system doesn't help here.

Not sure what this means.
 - Unintentional Ignored Expressions

dmd already flags dead code.
Feb 09 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

 safe guarantees memory safety, so these are non-issues in D.<

I think people want to use D instead of Java/C# for (when necessary) its lower level features, because if they don't need that extra little percentage of performance they probably want to use Java/C# in the first place, that have many more tools, software, cheap developers, etc. So D programs probably will contain some unsafe code. Of course in a D project a smart programmer tries to minimize the amount of unsafe lines of code. So a lint tool (or a better compiler) to spot those bugs in unsafe D code can be useful.
As you say, exceptions solve these problems.<

But there are many C++ programmers that don't use exceptions, for example they are not allowed in Google C++ code: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions So surely not 100% of D code will use exceptions. That's why I have said that D avoids some of those problems but not all them :-)
- Resource Leak: in theory the GC can help a lot here. In practice the D GC is
conservative, so it leaks by design.<<


It is a mistake to rely on the GC to free resources other than memory.
Properly, either RAII or scope guard statements should be used, as these do not
leak.<

I was saying that by design a conservative GC can leak memory, and I have seen that it actually does it if the programs allocates lot of memory.
- Uninitialized Values Read: D "solves" this problem initializing by default
all variables. In practice probably some default-initialized values can cause
troubles where a default initialization was not the right thing.<<


No analysis tool can help you if you initialize things to the wrong value.
That's very, very different from initializing things to GARBAGE and then using
those garbage values.<

What I meant is that I really like when the Java compiler tells me that a variable is not initialized :-)
I'll just state that turning on the optimizer will detect many cases of
dereferencing null pointers.<

Very good, and future D compilers can improve some more on this :-)
- Unsafe Use of Returned NULL: the D2 type system doesn't help here.<<


I meant that in D2 the type system doesn't support nonnullable references yet. Not-nullable references/pointers can't be null, so there's no need for the programmer to remember to test if they are null, so I think they can avoid some bugs caused by untested nulls. In practice I have never programmed with a language with this feature so I don't know what side effects it can have.
- Use Before Test: here the bugs are in the code paths of the try-except.<<


Thinking more about it, I think I was probably wrong. Thank you for your answers, as usual I am often wrong and you are usually right; I am not even remotely as good as you, but I try to help :-) Bye, bearophile
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
bearophile wrote:
 But there are many C++ programmers that don't use exceptions, for
 example they are not allowed in Google C++ code: 
 http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions
  So surely not 100% of D code will use exceptions. That's why I have
 said that D avoids some of those problems but not all them :-)

I don't know why Google proscribes exceptions, but exceptions still have an uneasy relationship with C++. Older compilers may not support them properly, much C++ code uses them incorrectly and inconsistently, there isn't good support for exception safety or transaction safety, being able to throw value types is a big mistake, etc.
Feb 09 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:
 I don't know why Google proscribes exceptions,

If you look at that link, and you click on the arrow, you will see a nice list of pro-cons :-) Bye, bearophile
Feb 09 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-09 19:26:05 -0500, bearophile <bearophileHUGS lycos.com> said:

 Walter Bright:
 I don't know why Google proscribes exceptions,

If you look at that link, and you click on the arrow, you will see a nice list of pro-cons :-)

And below the pro-cons it says they avoid exceptions because they have a lot of exception-unsafe code that would break if you were to throw exceptions, suggesting it might be otherwise if it wasn't for that legacy code. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Feb 09 2010
parent "Nick Sabalausky" <a a.a> writes:
"Michel Fortin" <michel.fortin michelf.com> wrote in message 
news:hkt3gq$19pp$1 digitalmars.com...
 On 2010-02-09 19:26:05 -0500, bearophile <bearophileHUGS lycos.com> said:

 Walter Bright:
 I don't know why Google proscribes exceptions,

If you look at that link, and you click on the arrow, you will see a nice list of pro-cons :-)

And below the pro-cons it says they avoid exceptions because they have a lot of exception-unsafe code that would break if you were to throw exceptions, suggesting it might be otherwise if it wasn't for that legacy code.

It doesn't just suggest it, it flat-out states it: "Things would probably be different if we had to do it all over again from scratch."
Feb 10 2010
prev sibling parent Justin Johansson <no spam.com> writes:
Walter Bright wrote:
 bearophile wrote:
 But there are many C++ programmers that don't use exceptions, for
 example they are not allowed in Google C++ code: 
 http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Exceptions
  So surely not 100% of D code will use exceptions. That's why I have
 said that D avoids some of those problems but not all them :-)

I don't know why Google proscribes exceptions, but exceptions still have an uneasy relationship with C++. Older compilers may not support them properly, much C++ code uses them incorrectly and inconsistently, there isn't good support for exception safety or transaction safety, being able to throw value types is a big mistake, etc.

If C++ exceptions are thrown and caught within, and are completely self- -contained within, a (C++) library then nobody should be the wiser. The only possible issue can be about whether the library should be allowed to propagate exceptions across the API boundary and into client code. The pro and con arguments should take the client code interaction / API context into consideration. Otherwise it's just a blatant, ill-considered consideration (ill-pun considered as well). Cheers Justin Johansson
Feb 10 2010
prev sibling next sibling parent reply sybrandy <sybrandy gmail.com> writes:
 - Use Before Test: here the bugs are in the code paths of the try-except.

Not sure what this means.

If I remember correctly from when I had an opportunity to look at these types of tools, this is a case where, in C or C++, you have a variable that may be uninitialized or set to NULL, but you don't test for it until you have tried to use it. I believe the root cause of this bug has nothing to do with the language, but with people going back into code they don't fully understand and trying to either A) use the variable and not know that the check is in there or B) they put the check in to "fix" a bug, but didn't put it in the correct spot. Needless to say, with D setting variables to reasonable defaults, I believe this is mitigated, but not entirely prevented.
Feb 09 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
sybrandy wrote:
 - Use Before Test: here the bugs are in the code paths of the 
 try-except.

Not sure what this means.

If I remember correctly from when I had an opportunity to look at these types of tools, this is a case where, in C or C++, you have a variable that may be uninitialized or set to NULL, but you don't test for it until you have tried to use it. I believe the root cause of this bug has nothing to do with the language, but with people going back into code they don't fully understand and trying to either A) use the variable and not know that the check is in there or B) they put the check in to "fix" a bug, but didn't put it in the correct spot.

That makes sense.
 Needless to say, with D setting variables to reasonable defaults, I 
 believe this is mitigated, but not entirely prevented.

You're right.
Feb 09 2010
prev sibling parent reply Brad Roberts <braddr bellevue.puremagic.com> writes:
On Tue, 9 Feb 2010, Walter Bright wrote:

  safe guarantees memory safety, so these are non-issues in D.
 
 Using  safe makes these non-issues.
 
 As you say, exceptions solve these problems.
 
 It is a mistake to rely on the GC to free resources other than memory.
 Properly, either RAII or scope guard statements should be used, as these do
 not leak.
 
 No analysis tool can help you if you initialize things to the wrong value.
 That's very, very different from initializing things to GARBAGE and then using
 those garbage values.
 
 We've certainly had long threads about this one. I'll just state that turning
 on the optimizer will detect many cases of dereferencing null pointers.
 
 dmd already flags dead code.

Nice black and white world you are acting like you live in there Walter. I know you know better, so why do you pretend otherwise? You're assuming perfect use of the language and largely simple code. Granted, there's a lot of things that D does better, and the tools in the language make it easier to write better code... but developers are corner cutters at heart. Additionally, not everything can be done in safe (or it wouldn't be an optional mode), and not everything can use RAII for resource management. Coverity, as has been discussed here in the past, is a very powerful tool that solves very real world problems... at an astronomical price. A sort-of close cousin to Coverity is Fortify. Fortify has a security slant to it, but is solving essentially the same class of problems. Compilers do, in concept, a lot of the things that Coverity does, but at a very different scale. Coverty is aimed at whole application long range effect analysis. It's got good tools for over-time analysis of builds to help ensure incremental improvements (ie, prevent quality slipping backwards). All in all, it's well engineered for large systems and large teams. I wish I had access to it for everthing I develop. Sadly, the cost is too high. Later, Brad
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Brad Roberts wrote:
 Nice black and white world you are acting like you live in there Walter.  
 I know you know better, so why do you pretend otherwise?

First off, yes, if you program in C style, despite using D, your code is vulnerable to many of the problems that Coverity detects.
 You're assuming perfect use of the language and largely simple code.  
 Granted, there's a lot of things that D does better, and the tools in the 
 language make it easier to write better code... but developers are corner 
 cutters at heart.  Additionally, not everything can be done in  safe (or 
 it wouldn't be an optional mode), and not everything can use RAII for 
 resource management.

Yes, I am relying on convention to a degree for D to be helpful, and I've often enough railed against relying on convention. So why the contradiction? Let's take the lock()/unlock() pairing problem. In C, you often wind up with a rat's nest of control flow that gets really hard to verify has the unlock() along all paths. Coverity is a win here. In C++, you can wrap the lock()/unlock() in a struct and use RAII, but that's as ugly as sin and many will eschew it, thus Coverity wins again. But in D, lock(); scope(exit) unlock(); and you're done. There's nothing left to analyze. Nothing makes you write D code that way, but I contend that it is easy to write it that way (which addresses the corner cutters), and it is easy for a code review to verify it. No tangle of logic must be traced. So the problem is reduced from a 10 to a 1. It doesn't go away, but the payback of Coverity for these cases is greatly reduced. (And the cost of Coverity isn't just the purchase price, it has significant administration and false positive costs.) About safe: a lot of Coverity is focussed on memory safety, for example it spends a lot of effort analyzing use of C string functions. With safe, the D compiler guarantees your code is memory safe. No false positives. The idea is to reduce the system parts of your code to as small a fraction as possible. Then, you aren't running a million lines of code through Coverity to check for memory safety, you're running only a few percent of it, and those few percent will be a lot easier to review.
 Compilers do, in concept, a lot of the things that Coverity does, but at a 
 very different scale.  Coverty is aimed at whole application long range 
 effect analysis.  It's got good tools for over-time analysis of builds to 
 help ensure incremental improvements (ie, prevent quality slipping 
 backwards).  All in all, it's well engineered for large systems and large 
 teams.
 
 I wish I had access to it for everthing I develop.  Sadly, the cost is too 
 high.

D has moved a lot towards supplying by default a lot of what Coverity claims to do. By making such an expensive tool irrelevant for D, we can make D much more cost effective.
Feb 09 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:
 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:
 
 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.

D doesn't provide non-nullable types

Yes, there have been a couple long threads about that. Dereferencing a null pointer is a bug, but not a security/safety issue.
 and local refs can still escape the scope easily.

Yes, that is a problem, and we've been trying to fix that.
Feb 10 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Walter Bright" <newshound1 digitalmars.com> wrote in message 
news:hkv1mg$s6e$1 digitalmars.com...
 retard wrote:
 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:

 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.

D doesn't provide non-nullable types

Yes, there have been a couple long threads about that. Dereferencing a null pointer is a bug, but not a security/safety issue.

Right, it's just not worthwhile for a language to provide help avoiding bugs.
Feb 10 2010
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Nick Sabalausky wrote:
 "Walter Bright" <newshound1 digitalmars.com> wrote in message 
 news:hkv1mg$s6e$1 digitalmars.com...
 retard wrote:
 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:

 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.


null pointer is a bug, but not a security/safety issue.

Right, it's just not worthwhile for a language to provide help avoiding bugs.

It's a benefit/cost issue. There are no bug-preventing features that are without cost, the idea is to maximize the ratio.
Feb 10 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Walter Bright:

[about non-nullable types]
 It's a benefit/cost issue. There are no bug-preventing features that are 
 without cost, the idea is to maximize the ratio.

It surely has a cost (you can see it in the Cyclone language: it's safe but quite fussy, so the costs for the programmer can be high enough to discourage its use), but I have not tried this feature in other languages, so it's not easy for me to measure its costs :-) Bye, bearophile
Feb 10 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"bearophile" <bearophileHUGS lycos.com> wrote in message 
news:hkv7u7$1a31$1 digitalmars.com...
 Walter Bright:

 [about non-nullable types]
 It's a benefit/cost issue. There are no bug-preventing features that are
 without cost, the idea is to maximize the ratio.

It surely has a cost (you can see it in the Cyclone language: it's safe but quite fussy, so the costs for the programmer can be high enough to discourage its use), but I have not tried this feature in other languages, so it's not easy for me to measure its costs :-)

I was assuming he meant cost of implementing that feature, but maybe you're right...?
Feb 10 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Nick Sabalausky wrote:
 I was assuming he meant cost of implementing that feature, but maybe you're 
 right...? 

I meant the cost to the user.
Feb 10 2010
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:
 What exactly is the cost here? The non-nullability invariant can be 
 checked on compile time. It incurs no runtime overhead. Also the notation 
 can be quite terse, as we have seen in Other Languages(tm).

If you want, you can create a template NonNullable that wraps an existing type, and try that out.
Feb 10 2010
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-10 17:29:09 -0500, Walter Bright <newshound1 digitalmars.com> said:

 retard wrote:
 What exactly is the cost here? The non-nullability invariant can be 
 checked on compile time. It incurs no runtime overhead. Also the 
 notation can be quite terse, as we have seen in Other Languages(tm).

If you want, you can create a template NonNullable that wraps an existing type, and try that out.

Just did that yesterday. It's probably not what most people expect though: it doesn't do static checking. But at least it catches null assignments early, before the null value breaks something elsewhere in the program. For your enjoyment: /** Struct to make object references and pointers unable to accept a null value. The initial value is still null, but after the first assignment the value can never become null again. Attempting to assign null will throw an IllegalNullAssignment error. */ struct NonNullable(T) { private T _nonNullableValue; alias _getNonNullableValue this; this(T value) { opAssign(value); } this(NonNullable!T value) { // FIXME: Not polymorphic opAssign(value); } void opAssign(V, string file = __FILE__, int line = __LINE__)(V newValue) { if (isNull(newValue)) IllegalNullAssignment.bailOut(file, line, "Attempt to assign null to " "non-nullable " ~ T.stringof ~ "."); _nonNullableValue = newValue; } T _getNonNullableValue() { assert(_nonNullableValue, "Attempt to access uninitialized non-nullable " ~ T.stringof ~ "."); return _nonNullableValue; } private { bool isNull(T)(T value) if (__traits(compiles, value._nonNullableValue)) { return isNull(value._nonNullableValue); } bool isNull(T)(T value) if (__traits(compiles, value is null)) { return value is null; } } } /** Error thrown when attempting to assign a null value to a non-nullable. */ class IllegalNullAssignment : Error { this(string message) { super(message); } private static void bailOut(string file, int line, in char[] msg) { throw new IllegalNullAssignment(text(file, '(', line, "): ", msg)); } } unittest { NonNullable!Object o; NonNullable!ClassInfo o2 = o.classinfo; NonNullable!TypeInfo o3; //NonNullable!Object o4 = o2; // FIXME: polymorphic NonNullable o = new Object; o = o2; try { o = o3; // should throw o.toString(); assert(0, "prior assignment should have thrown"); } catch (IllegalNullAssignment e) { // ok, error thrown. } } -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Feb 10 2010
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Michel Fortin wrote:
 unittest {
     NonNullable!Object o;

Here o is null.
     NonNullable!ClassInfo o2 = o.classinfo;
     NonNullable!TypeInfo o3;
     //NonNullable!Object o4 = o2; // FIXME: polymorphic NonNullable
     o = new Object;
     o = o2;
     try {
         o = o3; // should throw
         o.toString();
         assert(0, "prior assignment should have thrown");
     } catch (IllegalNullAssignment e) {
         // ok, error thrown.
     }
 }

Feb 10 2010
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
Ary Borenszweig wrote:
 Michel Fortin wrote:
 unittest {
     NonNullable!Object o;

Here o is null.

Sorry, just read the class comments. But if this can't be implemented without compiler support... then it's not of much use. :-(
Feb 10 2010
next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2010-02-10 22:54:40 -0500, Ary Borenszweig <ary esperanto.org.ar> said:

 Ary Borenszweig wrote:
 Michel Fortin wrote:
 unittest {
     NonNullable!Object o;

Here o is null.

Sorry, just read the class comments. But if this can't be implemented without compiler support... then it's not of much use. :-(

To recapitulate the documentation comment at the top of NonNullable: it is indeed null until the first assignment. It's non nullable in the sense that you can't make it null, but it still can be null if you assign nothing to it. It means that if you encounter a null value in a non-nullable you know it was never initialized. So the obvious place to look is where it should have been initialized. I really wonder how bad this would be in practice (I haven't used it much yet). Unfortunately, there isn't a better way to implement NonNullable at the moment. An option could be to throw immediately at construction, but constructors with no arguments are not available for structs, and I'm not sure it'd be so great anyway. There is also the issue of lack of polymorphism: a NonNullable!MyObject variable won't implicitly convert to NonNullable!Object, although I've been able to make it work for non-construction assignments using a template opAssign. So I agree with you that it's of limited utility. I think it can still be useful at times. What would be great is non-nullable by default, but that's another thing entirely. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Feb 10 2010
parent Ary Borenszweig <ary esperanto.org.ar> writes:
Michel Fortin wrote:
 On 2010-02-10 22:54:40 -0500, Ary Borenszweig <ary esperanto.org.ar> said:
 
 Ary Borenszweig wrote:
 Michel Fortin wrote:
 unittest {
     NonNullable!Object o;

Here o is null.

Sorry, just read the class comments. But if this can't be implemented without compiler support... then it's not of much use. :-(

To recapitulate the documentation comment at the top of NonNullable: it is indeed null until the first assignment. It's non nullable in the sense that you can't make it null, but it still can be null if you assign nothing to it. It means that if you encounter a null value in a non-nullable you know it was never initialized. So the obvious place to look is where it should have been initialized. I really wonder how bad this would be in practice (I haven't used it much yet).

That makes sense. :)
 Unfortunately, there isn't a better way to implement NonNullable at the 
 moment. An option could be to throw immediately at construction, but 
 constructors with no arguments are not available for structs, and I'm 
 not sure it'd be so great anyway.

Maybe in D2 you will be able to put disable to the default constructor of the struct, so you'll have to explicity assign something to it.
 There is also the issue of lack of polymorphism: a NonNullable!MyObject 
 variable won't implicitly convert to NonNullable!Object, although I've 
 been able to make it work for non-construction assignments using a 
 template opAssign.
 
 So I agree with you that it's of limited utility. I think it can still 
 be useful at times. What would be great is non-nullable by default, but 
 that's another thing entirely.

Yes, I think non-nullable by default is better.
Feb 11 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Ary Borenszweig:

But if this can't be implemented without compiler support... then it's not of
much use. :-(<

There's another solution: to make D2/D3 itself flexible enough that you can implement your own simple "extensions" to the type system, so you can actually implement not-nullable references by default. This isn't impossible (there are many papers about this, it can be done in Scheme, etc). Once such flexibility is present, you can quickly create such plug-ins to the type system. Like in Firefox, plug-ins are a very good way to experiment new features, that can be later added to the main software if they are good. (Recently GCC has gained the ability to be use plug-ins, but here I am talking about something in the language, not in the compiler). Related: I think the downsides of something like the AST macros (that is community fragmentation, etc) can be quite mitigated if the main purpose of them is to allow to create experimental features that later can be refused or accepted into the official language :-) Bye, bearophile
Feb 11 2010
parent reply dennis luehring <dl.soluz gmx.net> writes:
 There's another solution: to make D2/D3 itself flexible enough that you can
implement your own simple "extensions" to the type system, so you
can actually implement not-nullable references by default. This isn't 

it can be done in Scheme, etc).

"everything is possible with code" - its not the question of how to start an plugin/dll based function from descent compiler steps... thats too easy to even talk about but how should the plugin-system work? what compiler-functions should be changeable through plugins? how do the different engine like parser, semantic analyser etc work together with the plugin, etc.? that is the nearly impossible to answer question - what do the LLVM-guys think about plugin like that and firefox-plugins are far too easy in its possible feature-set compared to what is needed to add an type to an compiler
Feb 11 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
dennis luehring:
 but how should the plugin-system work?

I am ignorant still about type systems. But see something about pluggable type systems: http://lambda-the-ultimate.org/node/1311
what do the LLVM-guys think about plugin like that<

Chris Lattner, the boss of the LLVM project, has told me that he is not interested in a plug-in system for LLVM similar to the new GCC one, because LLVM has adopted kind of the opposite design strategy: it's made of loosely connected components that you can recombine and reuse to do what you want. Bye, bearophile
Feb 11 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
I've just found a nice article about this topic, "LCA: Static analysis with GCC
plugins", by Jonathan Corbet:
http://lwn.net/Articles/370717/

Beside allowing plug-ins, they have done something else to GCC:
The ability to attach attributes to objects in the compiled code makes it easy
to pass hints through to later processing steps. The new pass manager brings a
relatively modern structure to a compiler which did not originally have one.
And the GIMPLE intermediate representation provides much of the rest of what's
needed for code which needs to inspect other code.<

So now they are essentially extending the C++ type system using small JavaScript programs:
Various other tools have been written. The final.js script (a dozen lines of
code which can be seen on this page) looks for C++ methods tagged with the
"final" attribute; any attempt to override those methods will result in a
compilation error. It is, in other words, a port of the Java final keyword to
C++. A checker which might be interesting in other environments - including the
kernel - is flow.js, which can add a constraint that all exits from a function
must flow through a specific label.<

The final.js code: https://developer.mozilla.org/En/Dehydra/Using_Dehydra /** * Helper function: returns true if a class is marked with the "final" attribute. */ function isFinal(c) { if (!c.attributes) return false; for each (let a in c.attributes) if (a.name == 'user' && a.value == 'final') return true; return false; } function process_type(t) { if (t.bases) for each (let base in t.bases) if (isFinal(base.type)) error("class " + t.name + " extends final class " + base.type.name, t.loc); } Then in C++ the syntax of final class is not nice: class __attribute__((user("final"))) MyClass { In D you can probably write the same thing as (D already has final, so this is just to compare): final class MyClass { I think you can implement nonnullable references too (but you can't use the nice "?" syntax for nullables). Now please give me five minutes to dream about doing something similar using Python for D :-) It can be even possible to use D to write those scripts, especially once types become more first class values in D. Bye, bearophile
Feb 12 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
retard Wrote:
 That kind of code is rather clumsy.

That code is probably not the best possible way to express those constraints. But they have a good enough JavaScript JIT compiler, they know JavaScript well, so using JS for that purpose can be good enough [TM] and it can be better than not being able to do that. Bye, bearophile
Feb 12 2010
prev sibling parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
bearophile wrote:
 Then in C++ the syntax of final class is not nice:
 class __attribute__((user("final"))) MyClass {

To be pedantic, C++ does not have final classes. That's a compiler extension. Ali
Feb 12 2010
prev sibling parent "Nick Sabalausky" <a a.a> writes:
"retard" <re tard.com.invalid> wrote in message 
news:hkv5op$s4n$1 digitalmars.com...
 Wed, 10 Feb 2010 15:29:06 -0500, Nick Sabalausky wrote:

 "Walter Bright" <newshound1 digitalmars.com> wrote in message
 news:hkv1mg$s6e$1 digitalmars.com...
 Yes, there have been a couple long threads about that. Dereferencing a
 null pointer is a bug, but not a security/safety issue.

bugs.

Having such a language would cause unemployment among expensive professional programmers. Thus not worth it. That's why non-nullable types aren't even worth a try in an experimental branch of dmd unlike 1000 times more useful features like built-in compiler support for regular expressions, opFish or having an integrated email client.

opFish!!! Yes, I want it! :) Although that would beg the question of what would be the appropriate way to express gratitude for it if it were to get implemented: Option 1: Thanks for op the fish! Option 2: Thanks for all the opFish! Option 30: Icehouse (<--- Ok, that joke's probably *really* obscure)
Feb 10 2010
prev sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Leandro Lucarella wrote:
 This supposedly "safe" program under Mac OS X 10.6 doesn't give any error
 neither at compile time nor at runtime, yet it isn't memory-safe at all as
 it corrupts some part of the memory space.

safe is in its infancy, and I expect there to be some gaps. We'll get them plugged over time. In the meantime, it's great that you filed a bug report on it.
Feb 10 2010
prev sibling parent Leandro Lucarella <llucax gmail.com> writes:
Walter Bright, el 10 de febrero a las 14:28 me escribiste:
 Leandro Lucarella wrote:
This supposedly "safe" program under Mac OS X 10.6 doesn't give any error
neither at compile time nor at runtime, yet it isn't memory-safe at all as
it corrupts some part of the memory space.

safe is in its infancy, and I expect there to be some gaps. We'll get them plugged over time. In the meantime, it's great that you filed a bug report on it.

It wasn't me, but kudos for the reporter indeed =) -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ----------------------------------------------------------------------
Feb 11 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:

 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we can
 make D much more cost effective.

D doesn't provide non-nullable types and local refs can still escape the scope easily. There are several kinds of bugs that cause null pointer exceptions / segfaults. Now it took me 2 minutes to come up two examples: class foo { Object a; this(Object a) { a = a; } void bar() { a.toString(); } Object[] baz() { scope Object a = new Object; return [ a ]; } } void main() { auto a = new Object; // auto b = new foo(a); // segfault // b.baz()[0].toString(); // segfault }
Feb 10 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Wed, 10 Feb 2010 15:29:06 -0500, Nick Sabalausky wrote:

 "Walter Bright" <newshound1 digitalmars.com> wrote in message
 news:hkv1mg$s6e$1 digitalmars.com...
 retard wrote:
 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:

 D has moved a lot towards supplying by default a lot of what Coverity
 claims to do. By making such an expensive tool irrelevant for D, we
 can make D much more cost effective.

D doesn't provide non-nullable types

Yes, there have been a couple long threads about that. Dereferencing a null pointer is a bug, but not a security/safety issue.

bugs.

Having such a language would cause unemployment among expensive professional programmers. Thus not worth it. That's why non-nullable types aren't even worth a try in an experimental branch of dmd unlike 1000 times more useful features like built-in compiler support for regular expressions, opFish or having an integrated email client.
Feb 10 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Wed, 10 Feb 2010 12:47:41 -0800, Walter Bright wrote:

 Nick Sabalausky wrote:
 "Walter Bright" <newshound1 digitalmars.com> wrote in message
 news:hkv1mg$s6e$1 digitalmars.com...
 retard wrote:
 Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:

 D has moved a lot towards supplying by default a lot of what
 Coverity claims to do. By making such an expensive tool irrelevant
 for D, we can make D much more cost effective.


null pointer is a bug, but not a security/safety issue.

bugs.

It's a benefit/cost issue. There are no bug-preventing features that are without cost, the idea is to maximize the ratio.

What exactly is the cost here? The non-nullability invariant can be checked on compile time. It incurs no runtime overhead. Also the notation can be quite terse, as we have seen in Other Languages(tm).
Feb 10 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Wed, 10 Feb 2010 23:11:54 -0800, Walter Bright wrote:

 Nick Sabalausky wrote:
 I was assuming he meant cost of implementing that feature, but maybe
 you're right...?

I meant the cost to the user.

The fact is that some of the references in a correct program are non- nullable. If the program behaves according to the spec, those references will never point to null. What a language designer should do is to analyze the use cases and make the more common case the default. Type inference could aid the user here. I'm not sure how much trouble wrong guesses would cause, though.
Feb 11 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Thu, 11 Feb 2010 16:20:55 -0500, bearophile wrote:

 dennis luehring:
 but how should the plugin-system work?

I am ignorant still about type systems. But see something about pluggable type systems: http://lambda-the-ultimate.org/node/1311

In some educational Compiler Done Right (tm) projects the various parts of the compiler might be loosely coupled clean modules, but experience shows that in real life when you take e.g. efficiency into consideration, compilers tend to get nasty. Both parser and type system are two of the subsystems that almost everything else on a compiler depends on.
 
 
what do the LLVM-guys think about plugin like that<

Chris Lattner, the boss of the LLVM project, has told me that he is not interested in a plug-in system for LLVM similar to the new GCC one, because LLVM has adopted kind of the opposite design strategy: it's made of loosely connected components that you can recombine and reuse to do what you want. Bye, bearophile

Feb 11 2010
prev sibling parent retard <re tard.com.invalid> writes:
Fri, 12 Feb 2010 12:07:16 -0500, bearophile wrote:

 /**
  * Helper function: returns true if a class is marked with the "final"
  attribute. */
 function isFinal(c) {
   if (!c.attributes)
     return false;
   for each (let a in c.attributes)
     if (a.name == 'user' && a.value == 'final')
       return true;
   return false;
 }
 function process_type(t) {
   if (t.bases)
     for each (let base in t.bases)
       if (isFinal(base.type))
         error("class " + t.name + " extends final class " +
         base.type.name, t.loc);
 }

That kind of code is rather clumsy. Having written couple of toy languages, I really enjoy expressing the rules as immutable declarative axioms, e.g.: final_class_rule: node.metatype == class && node.hasBases ==> ! node.bases.exists(n => n.attributes.contains("final")) By using declarative notion, the ambiguity of rules can be tested and rules can be verified more easily.
Feb 12 2010
prev sibling parent Leandro Lucarella <llucax gmail.com> writes:
Walter Bright, el 10 de febrero a las 11:33 me escribiste:
 retard wrote:
Tue, 09 Feb 2010 18:49:31 -0800, Walter Bright wrote:

D has moved a lot towards supplying by default a lot of what Coverity
claims to do. By making such an expensive tool irrelevant for D, we can
make D much more cost effective.

D doesn't provide non-nullable types

Yes, there have been a couple long threads about that. Dereferencing a null pointer is a bug, but not a security/safety issue.

What about 3677? Quote: -------------------------------- This supposedly "safe" program under Mac OS X 10.6 doesn't give any error neither at compile time nor at runtime, yet it isn't memory-safe at all as it corrupts some part of the memory space. struct S { uint[100000] a; uint b; } S* s = null; safe void main() { s.b = 1; } This happen because the offset for member "b" is very far, allowing it to falls on another memory page which happen to exists. -------------------------------- http://d.puremagic.com/issues/show_bug.cgi?id=3677 It's a strange case, I know... -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ----------------------------------------------------------------------
Feb 10 2010