www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Non-null objects, the Null Object pattern, and T.init

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I were talking today about the null pointer issue and he had 
the following idea.

One common idiom to replace null pointer exceptions with milder 
reproducible errors is the null object pattern, i.e. there is one object 
that is used in lieu of the null reference to initialize all otherwise 
uninitialized references. In D that would translate naturally to:

class Widget
{
     private int x;
     private Widget parent;
     this(int y) { x = y; }
     ...
     // Here's the interesting part
     static Widget init = new Widget(42);
}

Currently the last line doesn't compile, but we can make it work if the 
respective constructor is callable during compilation. The compiler will 
allocate a static buffer for the "new"ed Widget object and will make 
init point there.

Whenever a Widget is to be default-initialized, it will point to 
Widget.init (i.e. it won't be null). This beautifully extends the 
language because currently (with no init definition) Widget.init is null.

So the init Widget will satisfy:

assert(x == 42 && parent is Widget.init);

Further avenues are opened by thinking what happens if e.g. init is 
private or  disable-d.

Thoughts?


Andrei
Jan 16 2014
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu 
wrote:
     static Widget init = new Widget(42);

The problem here is if it is in a mutable data segment: Widget i; // implicitly set to init assert(i is Widget.init); // uh oh i.mutate(); Widget i2; // set to widget init... assert(i is i2); // oh dear // i2 now reflects the mutation done above! And if the init is put in a read-only segment similar to a string buffer, we still get a segmentation fault when we try to mutate it. I'm not necessarily against having the option, but I think it would generally be more trouble than it is worth. If you have an object that you want to guarantee is not null, make the nullable implementation class Widget_impl {} then define alias Widget = NotNull!Widget_impl; and force initialization that way.
Jan 16 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/16/14 5:53 PM, Adam D. Ruppe wrote:
 On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu wrote:
     static Widget init = new Widget(42);

The problem here is if it is in a mutable data segment: Widget i; // implicitly set to init assert(i is Widget.init); // uh oh i.mutate(); Widget i2; // set to widget init... assert(i is i2); // oh dear // i2 now reflects the mutation done above!

Yah, that would be expected.
 I'm not necessarily against having the option, but I think it would
 generally be more trouble than it is worth. If you have an object that
 you want to guarantee is not null, make the nullable implementation

 class Widget_impl {}

 then define

 alias Widget = NotNull!Widget_impl;

 and force initialization that way.

Noted. Andrei
Jan 16 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/16/14 6:11 PM, Adam D. Ruppe wrote:
 On Friday, 17 January 2014 at 02:04:27 UTC, Andrei Alexandrescu wrote:
 Yah, that would be expected.

Yeah, but I think people would find it weird. This kind of thing is actually possible today: class Foo { } class Bar { Foo foo = new Foo(); // this gets a static reference into the typeinfo.init (since new Foo is evaluated at compile time!) which is blitted over... }

That wouldn't be allowed. Andrei
Jan 16 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/16/2014 6:11 PM, Adam D. Ruppe wrote:
 I think people would be a bit surprised if it was "Foo foo;"

We really don't want any foo-foo stuff in D.
Jan 16 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/17/2014 03:11 AM, Adam D. Ruppe wrote:
 On Friday, 17 January 2014 at 02:04:27 UTC, Andrei Alexandrescu wrote:
 Yah, that would be expected.

Yeah, but I think people would find it weird. This kind of thing is actually possible today:

class Foo{ int x = 2; } class Bar{ auto foo = new Foo(); } void main(){ auto ibar = new immutable(Bar)(); auto bar = new Bar(); static assert(is(typeof(ibar.foo.x)==immutable)); assert(ibar.foo.x==2); bar.foo.x=3; assert(ibar.foo.x==3); // uh-oh! }
Jan 16 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-01-17 07:18, Timon Gehr wrote:

 class Foo{ int x = 2; }

 class Bar{ auto foo = new Foo(); }

 void main(){
      auto ibar = new immutable(Bar)();
      auto bar = new Bar();
      static assert(is(typeof(ibar.foo.x)==immutable));
      assert(ibar.foo.x==2);
      bar.foo.x=3;
      assert(ibar.foo.x==3); // uh-oh!
 }

Oh, that's just bad. Yet another hole in the type system. -- /Jacob Carlborg
Jan 17 2014
prev sibling next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu 
wrote:
 Walter and I were talking today about the null pointer issue 
 and he had the following idea.

 One common idiom to replace null pointer exceptions with milder 
 reproducible errors is the null object pattern, i.e. there is 
 one object that is used in lieu of the null reference to 
 initialize all otherwise uninitialized references. In D that 
 would translate naturally to:

 class Widget
 {
     private int x;
     private Widget parent;
     this(int y) { x = y; }
     ...
     // Here's the interesting part
     static Widget init = new Widget(42);
 }

 Currently the last line doesn't compile, but we can make it 
 work if the respective constructor is callable during 
 compilation. The compiler will allocate a static buffer for the 
 "new"ed Widget object and will make init point there.

 Whenever a Widget is to be default-initialized, it will point 
 to Widget.init (i.e. it won't be null). This beautifully 
 extends the language because currently (with no init 
 definition) Widget.init is null.

 So the init Widget will satisfy:

 assert(x == 42 && parent is Widget.init);

 Further avenues are opened by thinking what happens if e.g. 
 init is private or  disable-d.

 Thoughts?


 Andrei

I am assuming init will be a property static function. So essentially we would be removing .init support for classes in compiler and pushing it out into Object. I would rather if that is an option an init static function, that it would be required to have override otherwise it would be too 'magical' for me atleast. The default can still be null.
Jan 16 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/16/14 5:57 PM, Rikki Cattermole wrote:
 I am assuming init will be a property static function.

In the example above it's just a static data member.
 So essentially we
 would be removing .init support for classes in compiler and pushing it
 out into Object.

 I would rather if that is an option an init static function, that it
 would be required to have override otherwise it would be too 'magical'
 for me atleast.
 The default can still be null.

Override only works for nonstatic methods. Andrei
Jan 16 2014
prev sibling next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu 
wrote:
 Whenever a Widget is to be default-initialized, it will point 
 to Widget.init (i.e. it won't be null). This beautifully 
 extends the language because currently (with no init 
 definition) Widget.init is null.

Hmm, what about derived classes? How do you check for a valid Widget given a DerivedWidget.init? class DerivedWidget : Widget { static DerivedWidget init = new DerivedWidget(...); } bool valid(Widget w) { return w !is Widget.init; } DerivedWidget foo; assert(!valid(foo)); // doesn't fire, foo is valid? The nice thing about null is that it is the bottom type, so it is a universal sentinel value. Also, what about interfaces? You cannot define an init for an interface. Obviously that could just be a known and accepted limitation of this proposal, but I'm not a huge fan of solutions that only work in a subset of situations. Perhaps there is a solution that I haven't thought of.
 assert(x == 42 && parent is Widget.init);

Is that meant to say "x is Widget.init"?
Jan 16 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/16/14 6:02 PM, Peter Alexander wrote:
 On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu wrote:
 Whenever a Widget is to be default-initialized, it will point to
 Widget.init (i.e. it won't be null). This beautifully extends the
 language because currently (with no init definition) Widget.init is null.

Hmm, what about derived classes? How do you check for a valid Widget given a DerivedWidget.init? class DerivedWidget : Widget { static DerivedWidget init = new DerivedWidget(...); } bool valid(Widget w) { return w !is Widget.init; } DerivedWidget foo; assert(!valid(foo)); // doesn't fire, foo is valid? The nice thing about null is that it is the bottom type, so it is a universal sentinel value.

This is a good point.
 Also, what about interfaces? You cannot define an init for an interface.
 Obviously that could just be a known and accepted limitation of this
 proposal, but I'm not a huge fan of solutions that only work in a subset
 of situations. Perhaps there is a solution that I haven't thought of.


 assert(x == 42 && parent is Widget.init);

Is that meant to say "x is Widget.init"?

To clarify: assert(Widget.init.x == 42 && Widget.init.parent is Widget.init); Andrei
Jan 16 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/16/14 6:13 PM, Peter Alexander wrote:
 However, I'm not sure this is good. Those kinds of reference cycles can
 easily lead to infinite loops. Is that really an improvement on a
 null-pointer dereference?

You don't have to use the null object pattern for all classes. It's opt-in. Andrei
Jan 16 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 17 January 2014 at 01:57:24 UTC, Rikki Cattermole 
wrote:
 I am assuming init will be a property static function. So 
 essentially we would be removing .init support for classes in 
 compiler and pushing it out into Object.

I don't think a function would work without major work since that would mean adding non-blit* default construction to the language - which isn't present at all right now.
Jan 16 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu
wrote:
 Walter and I were talking today about the null pointer issue 
 and he had the following idea.

 One common idiom to replace null pointer exceptions with milder 
 reproducible errors is the null object pattern, i.e. there is 
 one object that is used in lieu of the null reference to 
 initialize all otherwise uninitialized references. In D that 
 would translate naturally to:

 class Widget
 {
     private int x;
     private Widget parent;
     this(int y) { x = y; }
     ...
     // Here's the interesting part
     static Widget init = new Widget(42);
 }

 Currently the last line doesn't compile, but we can make it 
 work if the respective constructor is callable during 
 compilation. The compiler will allocate a static buffer for the 
 "new"ed Widget object and will make init point there.

 Whenever a Widget is to be default-initialized, it will point 
 to Widget.init (i.e. it won't be null). This beautifully 
 extends the language because currently (with no init 
 definition) Widget.init is null.

 So the init Widget will satisfy:

 assert(x == 42 && parent is Widget.init);

 Further avenues are opened by thinking what happens if e.g. 
 init is private or  disable-d.

 Thoughts?


 Andrei

Most object don't have a sensible init value. That is just hiding the problem under the carpet.
Jan 16 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/16/14 6:09 PM, Peter Alexander wrote:
 On Friday, 17 January 2014 at 02:07:08 UTC, deadalnix wrote:
 Most object don't have a sensible init value. That is just hiding
 the problem under the carpet.

That's true. class Widget { Widget parent; static Widget init = ??? } How do you define init?

Depends on what you want. Could be null or could have itself as a parent. The null object pattern is what it is. Andrei
Jan 16 2014
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 17 January 2014 at 02:07:08 UTC, deadalnix wrote:
 Most object don't have a sensible init value. That is just 
 hiding
 the problem under the carpet.

That's true. class Widget { Widget parent; static Widget init = ??? } How do you define init?
Jan 16 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 17 January 2014 at 02:04:27 UTC, Andrei Alexandrescu 
wrote:
 Yah, that would be expected.

Yeah, but I think people would find it weird. This kind of thing is actually possible today: class Foo { } class Bar { Foo foo = new Foo(); // this gets a static reference into the typeinfo.init (since new Foo is evaluated at compile time!) which is blitted over... } void main() { auto bar = new Bar(); // bar.foo is blitted to point at the static Foo auto bar2 = new Bar(); // and same thing assert(bar.foo is bar2.foo); // passes, but that's kinda weird } Granted, maybe the weirdness here is because the variable isn't static, so people expect it to be different, but I saw at least one person on the chat room be very surprised by this - and I was too, until I thought about how CTFE and Classinfo.init is implemented, then it made sense, but at first glance it was a bit weird. I think people would be a bit surprised if it was "Foo foo;" as well using the proposed .init thingy.
Jan 16 2014
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Friday, 17 January 2014 at 02:08:07 UTC, Andrei Alexandrescu 
wrote:
 assert(x == 42 && parent is Widget.init);

Is that meant to say "x is Widget.init"?

To clarify: assert(Widget.init.x == 42 && Widget.init.parent is Widget.init);

Ah right, missed the parent parameter... Also ignore my previous comment in that case :-) However, I'm not sure this is good. Those kinds of reference cycles can easily lead to infinite loops. Is that really an improvement on a null-pointer dereference?
Jan 16 2014
prev sibling next sibling parent "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Friday, 17 January 2014 at 02:06:03 UTC, Andrei Alexandrescu 
wrote:
 I would rather if that is an option an init static function, 
 that it
 would be required to have override otherwise it would be too 
 'magical'
 for me atleast.
 The default can still be null.

Override only works for nonstatic methods.

I know it doesn't. Its more for making it nicer to read than anything. An actual property as you've shown would be nice. But having another place which can be abused, which you can write code for is nice.
Jan 16 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
deadalnix:

 Most object don't have a sensible init value. That is just 
 hiding the problem under the carpet.

If there's desire to solve this problem I think that improving the type system to avoid nulls where they are not desired is better than having an init object. So aren't not-nullable pointers and references a better solution? Bye, bearophile
Jan 16 2014
prev sibling next sibling parent "inout" <inout gmail.com> writes:
On Friday, 17 January 2014 at 02:52:15 UTC, bearophile wrote:
 deadalnix:

 Most object don't have a sensible init value. That is just 
 hiding the problem under the carpet.

If there's desire to solve this problem I think that improving the type system to avoid nulls where they are not desired is better than having an init object. So aren't not-nullable pointers and references a better solution? Bye, bearophile

This! Also, if anything, it's better to turn `init` into a method rather than an object. The following would work all of a sudden: class Foo { Bar bar = new Bar(); int i = 42; Foo() { assert(bar !is null); assert(i == 42); } // auto-generated private final void init(Foo foo) { foo.bar = new Bar(); foo.i = 42; } }
Jan 16 2014
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-17 01:42:37 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Walter and I were talking today about the null pointer issue and he had 
 the following idea.
 
 [...]
 
 Thoughts?

Which null pointer issue were you discussing exactly? The one I'm mostly concerned about is the propagation of a null value from point A to point B in a program, where you only detect the null value at point B through a null dereference making it hard to figure out from where this unexpected null value comes from. Replace 'null' with 'sentinel' and it's no easier to figure out where the invalid value comes from. Except now instead of checking for null you have to check for null *and* for T.init *and* for all the various Sublcass.init you might get. The idea is interesting, but I see it creating more problems than it would solve. I'm not even sure I understand what problem it tries to solve. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 16 2014
next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 17 Jan 2014 05:29:05 -0000, H. S. Teoh <hsteoh quickfur.ath.cx>  
wrote:
 Now, if we modify this sentinel to instead record the location of the
 code that first initialized it (via __FILE__ and __LINE__ default
 parameters perhaps), then we can set it up to print out this information
 at a convenient juncture, so that the source of the uninitialized
 reference can be determined. *Then* perhaps it will be a start of a
 solution to this issue. (Though it still has limitations in the sense
 that the problem can only be caught at runtime, whereas some cases of
 null dereference preferably should be caught at compile-time.)

So.. if we had a base class for all objects which obtained the file and line when created by assignment (from init) and threw on any dereference (opDispatch) that would do it, right? R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Jan 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 17, 2014 at 12:51:38PM -0000, Regan Heath wrote:
 On Fri, 17 Jan 2014 05:29:05 -0000, H. S. Teoh
 <hsteoh quickfur.ath.cx> wrote:
Now, if we modify this sentinel to instead record the location of the
code that first initialized it (via __FILE__ and __LINE__ default
parameters perhaps), then we can set it up to print out this
information at a convenient juncture, so that the source of the
uninitialized reference can be determined. *Then* perhaps it will be
a start of a solution to this issue. (Though it still has limitations
in the sense that the problem can only be caught at runtime, whereas
some cases of null dereference preferably should be caught at
compile-time.)

So.. if we had a base class for all objects which obtained the file and line when created by assignment (from init) and threw on any dereference (opDispatch) that would do it, right?

If Andrei's proposal were extended so that .init can be overridden by a member *function*, then this would work: class NullTracker { override typeof(this) init(string _file=__FILE__, size_t _line = __LINE__) { class Impl : NullTracker { string file; size_t line; this(string f, size_t l) { file=f; line=l; } override void method1() { nullDeref(); } override void method2() { nullDeref(); } ... void nullDeref() { // N.B.: puts the *source* of the // null in the Exception. throw new Exception( "Null dereference", file, line); } } return new Impl(_file, _line); } void method1() {} void method2() {} ... } T -- People walk. Computers run.
Jan 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 17, 2014 at 10:00:26AM -0800, H. S. Teoh wrote:
[...]
 If Andrei's proposal were extended so that .init can be overridden by a
 member *function*, then this would work:
 
 	class NullTracker {
 		override typeof(this) init(string _file=__FILE__,
 						size_t _line = __LINE__)
 		{
 			class Impl : NullTracker {
 				string file;
 				size_t line;
 				this(string f, size_t l) { file=f; line=l; }
 
 				override void method1() { nullDeref(); }
 				override void method2() { nullDeref(); }
 				...
 				void nullDeref() {
 					// N.B.: puts the *source* of the
 					// null in the Exception.
 					throw new Exception(
 						"Null dereference",
 						file, line);

P.S. A better message might be "Null dereference, uninitialized object from %s:%d", to distinguish the site of the null dereference vs. its ultimate source. T -- "How are you doing?" "Doing what?"
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 17 January 2014 at 05:30:45 UTC, H. S. Teoh wrote:
 Now, if we modify this sentinel to instead record the location 
 of the
 code that first initialized it (via __FILE__ and __LINE__ 
 default
 parameters perhaps), then we can set it up to print out this 
 information
 at a convenient juncture, so that the source of the 
 uninitialized
 reference can be determined. *Then* perhaps it will be a start 
 of a

Which is easy to do (in theory). Page zero is over a thousand unique adresses. All you have to do is to convert null-tests to a range test. E.g. (addr&MASK)==0
Jan 19 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 19, 2014 at 09:41:20PM +0000, digitalmars-d-bounces puremagic.com
wrote:
 On Friday, 17 January 2014 at 05:30:45 UTC, H. S. Teoh wrote:
Now, if we modify this sentinel to instead record the location of the
code that first initialized it (via __FILE__ and __LINE__ default
parameters perhaps), then we can set it up to print out this
information at a convenient juncture, so that the source of the
uninitialized reference can be determined. *Then* perhaps it will be
a start of a

Which is easy to do (in theory). Page zero is over a thousand unique adresses. All you have to do is to convert null-tests to a range test. E.g. (addr&MASK)==0

It's not that simple. In assembly, usually the pointer value is just a base address, you add the field offset on top of that before you even attempt to load anything from memory. So you can't use "different nulls" to represent different source locations that easily. T -- There's light at the end of the tunnel. It's the oncoming train.
Jan 19 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 20 January 2014 at 01:22:34 UTC, H. S. Teoh wrote:
 adresses. All you have to do is to convert null-tests to a 
 range
 test. E.g. (addr&MASK)==0

It's not that simple. In assembly, usually the pointer value is just a base address, you add the field offset on top of that before you even attempt to load anything from memory. So you can't use "different nulls" to represent different source locations that easily.

Why not? You don't trap address zero, the MMU traps pages of 4K size on x86. But when you have an explicit nullptr test you will have to mask it before testing the zero-flag in the control register.
Jan 20 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 20 January 2014 at 12:20:58 UTC, Ola Fosheim Grøstad 
wrote:
 But when you have an explicit nullptr test you will have to 
 mask it before testing the zero-flag in the control register.

And just to make it explicit: you will have to add the masking logic to all comparisons of nullable pointers too if you want to allow comparison of null to be valid. aptr==bptr => ((aptr&MASK==0)&&(bptr&MASK==0))||aptr==bptr or something like that.
Jan 20 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jan 16, 2014 at 11:00:24PM -0500, Michel Fortin wrote:
 On 2014-01-17 01:42:37 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:
 
Walter and I were talking today about the null pointer issue and
he had the following idea.

[...]

Thoughts?

Which null pointer issue were you discussing exactly? The one I'm mostly concerned about is the propagation of a null value from point A to point B in a program, where you only detect the null value at point B through a null dereference making it hard to figure out from where this unexpected null value comes from. Replace 'null' with 'sentinel' and it's no easier to figure out where the invalid value comes from. Except now instead of checking for null you have to check for null *and* for T.init *and* for all the various Sublcass.init you might get. The idea is interesting, but I see it creating more problems than it would solve. I'm not even sure I understand what problem it tries to solve.

AFAICT the null object pattern is intended to prevent dereferencing a null pointer (i.e., prevent a segfault on Posix), by providing a dummy object that has no-op stubs for methods and default values for fields. But from what I've been observing, what people have been asking for on this forum has been more a way of *preventing* null (or a sentinel, as here) from being assigned to a reference to begin with. So as far as that is concerned, this proposal doesn't really address the issue. Now, if we modify this sentinel to instead record the location of the code that first initialized it (via __FILE__ and __LINE__ default parameters perhaps), then we can set it up to print out this information at a convenient juncture, so that the source of the uninitialized reference can be determined. *Then* perhaps it will be a start of a solution to this issue. (Though it still has limitations in the sense that the problem can only be caught at runtime, whereas some cases of null dereference preferably should be caught at compile-time.) T -- Music critic: "That's an imitation fugue!"
Jan 16 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/16/2014 5:42 PM, Andrei Alexandrescu wrote:
 Walter and I were talking today about the null pointer issue and he had the
 following idea.

 One common idiom to replace null pointer exceptions with milder reproducible
 errors is the null object pattern, i.e. there is one object that is used in
lieu
 of the null reference to initialize all otherwise uninitialized references. In
D
 that would translate naturally to:

 class Widget
 {
      private int x;
      private Widget parent;
      this(int y) { x = y; }
      ...
      // Here's the interesting part
      static Widget init = new Widget(42);
 }

I was thinking of: property static Widget init() { ... }
Jan 16 2014
parent Artur Skawina <art.08.09 gmail.com> writes:
On 01/17/14 06:33, Walter Bright wrote:
 On 1/16/2014 5:42 PM, Andrei Alexandrescu wrote:
 Walter and I were talking today about the null pointer issue and he had the
 following idea.

 One common idiom to replace null pointer exceptions with milder reproducible
 errors is the null object pattern, i.e. there is one object that is used in
lieu
 of the null reference to initialize all otherwise uninitialized references. In
D
 that would translate naturally to:

 class Widget
 {
      private int x;
      private Widget parent;
      this(int y) { x = y; }
      ...
      // Here's the interesting part
      static Widget init = new Widget(42);
 }

I was thinking of: property static Widget init() { ... }

default() { ... } because a) you've just reinvented default construction; b) overloading 'init' like that is a bad idea (should this 'init()' be called before invoking a "normal" ctor?...) artur
Jan 17 2014
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01/17/2014 02:42 AM, Andrei Alexandrescu wrote:
 class Widget
 {
      private int x;
      private Widget parent;
      this(int y) { x = y; }
      ...
      // Here's the interesting part
      static Widget init = new Widget(42);
 }

 Currently the last line doesn't compile, but we can make it work if the
 respective constructor is callable during compilation. The compiler will
 allocate a static buffer for the "new"ed Widget object and will make
 init point there.

immutable(Widget) iwidget; // ?
Jan 16 2014
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-17 02:42, Andrei Alexandrescu wrote:

 Thoughts?

As others have said, I don't really see the point in this. It just replaces the existing sentinel value (null) with a different one. -- /Jacob Carlborg
Jan 17 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 17 January 2014 at 03:02:57 UTC, inout wrote:
 On Friday, 17 January 2014 at 02:52:15 UTC, bearophile wrote:
 deadalnix:

 Most object don't have a sensible init value. That is just 
 hiding the problem under the carpet.

If there's desire to solve this problem I think that improving the type system to avoid nulls where they are not desired is better than having an init object. So aren't not-nullable pointers and references a better solution? Bye, bearophile

This! Also, if anything, it's better to turn `init` into a method rather than an object. The following would work all of a sudden: class Foo { Bar bar = new Bar(); int i = 42; Foo() { assert(bar !is null); assert(i == 42); } // auto-generated private final void init(Foo foo) { foo.bar = new Bar(); foo.i = 42; } }

nothing. :)
Jan 17 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Friday, 17 January 2014 at 08:13:05 UTC, Namespace wrote:
 On Friday, 17 January 2014 at 03:02:57 UTC, inout wrote:
 On Friday, 17 January 2014 at 02:52:15 UTC, bearophile wrote:
 deadalnix:

 Most object don't have a sensible init value. That is just 
 hiding the problem under the carpet.

If there's desire to solve this problem I think that improving the type system to avoid nulls where they are not desired is better than having an init object. So aren't not-nullable pointers and references a better solution? Bye, bearophile

This! Also, if anything, it's better to turn `init` into a method rather than an object. The following would work all of a sudden: class Foo { Bar bar = new Bar(); int i = 42; Foo() { assert(bar !is null); assert(i == 42); } // auto-generated private final void init(Foo foo) { foo.bar = new Bar(); foo.i = 42; } }

nothing. :)

But IMO even better would be something like this: ---- class A { int id; this(int id) { this.id = id; } static A init() { return new A(42); } } A a; /// <-- A a = A.init; --> A a = new A(42); ---- Define your own init method which initialize the object to not null.
Jan 17 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu 
wrote:
 Walter and I were talking today about the null pointer issue 
 and he had the following idea.

 One common idiom to replace null pointer exceptions with milder 
 reproducible errors is the null object pattern, i.e. there is 
 one object that is used in lieu of the null reference to 
 initialize all otherwise uninitialized references. In D that 
 would translate naturally to:

 class Widget
 {
     private int x;
     private Widget parent;
     this(int y) { x = y; }
     ...
     // Here's the interesting part
     static Widget init = new Widget(42);
 }

 Currently the last line doesn't compile, but we can make it 
 work if the respective constructor is callable during 
 compilation. The compiler will allocate a static buffer for the 
 "new"ed Widget object and will make init point there.

 Whenever a Widget is to be default-initialized, it will point 
 to Widget.init (i.e. it won't be null). This beautifully 
 extends the language because currently (with no init 
 definition) Widget.init is null.

 So the init Widget will satisfy:

 assert(x == 42 && parent is Widget.init);

 Further avenues are opened by thinking what happens if e.g. 
 init is private or  disable-d.

 Thoughts?


 Andrei

Throwing out some more ideas, though non of this compiles right now (and I managed to make the compiler ICE). class Widget { private int x; this(int y) { x = y; } public property int halfOfX() { return x / 2; } public void printHello() { std.stdio.writeln("Hello"); } //Error: 'this' is only defined in non-static member functions, //not Widget static Widget init = new class Widget { disable public override property int halfOfX(); disable public override void printHello(); this() { super(int.init); } }; } Another idea that unfortunately doesn't work due to the way static assert works: class Widget { private int x; this(int y) { x = y; } public property int halfOfX() { return x / 2; } public void printHello() { std.stdio.writeln("Hello"); } static Widget init = new class Widget { //Error: static assert "Tried to access halfOfX //property of null Widget" override property int halfOfX() { static assert(false, "Tried to access halfOfX property of null Widget"); } override void printHello() { static assert(false, "Tried to access halfOfX property of null Widget"); } this() { super(int.init); } }; } Is there any way to get around this? And then there's this, which causes an ICE: class Widget { private int x; this(int y) { x = y; } public property int halfOfX() { return x / 2; } public void printHello() { std.stdio.writeln("Hello"); } static Widget init = new class Widget { this() { } }; }
Jan 17 2014
prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu 
wrote:
 One common idiom to replace null pointer exceptions with milder 
 reproducible errors is the null object pattern, i.e. there is

Usually null failures are hard to track when some function returned a null value as an indication of error when the programmer expected an exception. In that case early failure is better. I guess it cold make sense with "weak" pointers if they are wiped by the GC. Like if you delete a User object, then all "weak" pointers are set to a UnknownUser default object. So that you can call user.isLoggedIn() and get false, but you should be able to default to a different Class to get a different set of virtual calls. I could also see the use-case for lazy allocation. Basically having memory-cheap "smart" pointers that are checked when you trap zero-page access and automatically alloc/init an empty object before recovering. But the overhead is large. Not sure if init() is the right way to do it though, because you might need something more flexible. E.g. maybe you want access to the node that contained the offending null-pointer.
Jan 17 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:06 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu wrote:
 One common idiom to replace null pointer exceptions with milder reproducible
 errors is the null object pattern, i.e. there is

Usually null failures are hard to track when some function returned a null value as an indication of error when the programmer expected an exception.

I've almost never had a problem tracking down the cause of a null pointer. Usually just a few minutes with a debugger and getting a backtrace.
Jan 17 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 1:10 PM, H. S. Teoh wrote:
 On Fri, Jan 17, 2014 at 11:43:59AM -0800, Walter Bright wrote:
 [...]
 I've almost never had a problem tracking down the cause of a null
 pointer. Usually just a few minutes with a debugger and getting a
 backtrace.

I think this depends on the kind of code you write. In callback-heavy code, usually when you're multiplexing between many simultaneous request/response chains, these kinds of problems are very hard to track down. You'll see the null pointer somewhere in your callback's context structure, but no amount of backtrace will help you go any further because they all end at the event dispatch loop pretty shortly up the stack, which doesn't tell you where in the chain of events the null came from.

What you do then is go back as far as practical, then put asserts in.
 When there are n possible ancestors for each step in the chain, you're
 talking about n^k possible paths to investigate before you find the
 ultimate culprit. A few minutes is not going to suffice, even for
 moderate values of k.

Put asserts in, maybe a printf too. I do it all the time to track down problems. Memory corruption can be a difficult one to track down, but I've never had trouble tracking down a null pointer's source. They aren't any harder than any "I have a bad value in this field, I wonder who set it" problem. I.e. it isn't special.
 Nipping the null at the bud (i.e., know where in the code it was first
 set as null) is a real life-saver in these kinds of situations.

It's better for any bug to catch it at compile time.
Jan 17 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 2:05 PM, H. S. Teoh wrote:
 Of course, but that requires a few rounds of recompilation, repacking
 the EEPROM image, installation on device, and repeating a user
 interaction that possibly only randomly hits the bug. Hardly the work of
 a few minutes.

I've done EEPROM programming & debugging, I know how to do it. Nulls still aren't a special kind of bug. BTW, even back in the 1970's, people developing EEPROM systems would plug a special device into the board which would redirect the EEPROM access to some off-board RAM, or would debug using emulators. Maybe this technology has been forgotten :-)
 But in many cases, the source of a bad value is easy to find because of
 its uniqueness (hmm, why does variable x have a value of 24576? oh I
 know, search for 24576 in the code). Nulls, though, can occur in many
 places, and in D they are implicit. There's no easy way to search for
 that.

Come on. Every type in D has a default initializer. There's still nothing special about null. Even if you got rid of all the nulls and instead use the null object pattern, you're not going to find it any easier to track it down, and you're in even worse shape because now it can fail and you may not even detect the failure, or may discover the error much, much further from the source of the bug.
Jan 17 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 3:37 PM, Walter Bright wrote:
 Come on. Every type in D has a default initializer. There's still
 nothing special about null.

There is. Null pointers cause the process to finish.
 Even if you got rid of all the nulls and instead use the null object
 pattern, you're not going to find it any easier to track it down, and
 you're in even worse shape because now it can fail and you may not even
 detect the failure, or may discover the error much, much further from
 the source of the bug.

How do you know all that? Andrei
Jan 17 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 4:17 PM, Andrei Alexandrescu wrote:
 On 1/17/14 3:37 PM, Walter Bright wrote:
 Come on. Every type in D has a default initializer. There's still
 nothing special about null.

There is. Null pointers cause the process to finish.

This particular subthread is about finding the reason why there was a null, not about what happens when the invalid value is detected.
 Even if you got rid of all the nulls and instead use the null object
 pattern, you're not going to find it any easier to track it down, and
 you're in even worse shape because now it can fail and you may not even
 detect the failure, or may discover the error much, much further from
 the source of the bug.

How do you know all that?

Because I've tracked down the cause of many, many null pointers, and I've tracked down the cause of many, many other kinds of invalid values in a variable. Null pointers tend to get detected much sooner, hence closer to where they were set.
Jan 17 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 5:05 PM, Walter Bright wrote:
 On 1/17/2014 4:17 PM, Andrei Alexandrescu wrote:
 Even if you got rid of all the nulls and instead use the null object
 pattern, you're not going to find it any easier to track it down, and
 you're in even worse shape because now it can fail and you may not even
 detect the failure, or may discover the error much, much further from
 the source of the bug.

How do you know all that?

Because I've tracked down the cause of many, many null pointers, and I've tracked down the cause of many, many other kinds of invalid values in a variable. Null pointers tend to get detected much sooner, hence closer to where they were set.

I'm not sure at all. Don't forget you have worked on a category of programs for the last 15 years. Reactive/callback-based code can be quite different, as a poster noted. I'll check with the folks to confirm that. The larger point I'm trying to make is your position and mine requires we need to un-bias ourselves as much as we can. Andrei
Jan 17 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:48 PM, Andrei Alexandrescu wrote:
 On 1/17/14 5:05 PM, Walter Bright wrote:
 On 1/17/2014 4:17 PM, Andrei Alexandrescu wrote:
 Even if you got rid of all the nulls and instead use the null object
 pattern, you're not going to find it any easier to track it down, and
 you're in even worse shape because now it can fail and you may not even
 detect the failure, or may discover the error much, much further from
 the source of the bug.

How do you know all that?

Because I've tracked down the cause of many, many null pointers, and I've tracked down the cause of many, many other kinds of invalid values in a variable. Null pointers tend to get detected much sooner, hence closer to where they were set.

I'm not sure at all. Don't forget you have worked on a category of programs for the last 15 years. Reactive/callback-based code can be quite different, as a poster noted. I'll check with the folks to confirm that.

We've talked about the floating point issue, where nan values "poison" the results rather than raise a seg fault. I remembered later that a while back Don tried to change D to throw an exception when a floating point nan was encountered, specifically because this made it easier for him to track down the source of the nan. He said it was pretty hard to backtrack it from the eventual output. In any case, I'm open to data that supports the notion that delayed detection of an error makes the source of the error easier to identify. That's not been my experience, and my intuition also finds that improbable.
 The larger point I'm trying to make is your position and mine requires we need
 to un-bias ourselves as much as we can.

I agree with that. So I await data :-)
Jan 17 2014
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01/18/2014 12:37 AM, Walter Bright wrote:
 Come on. Every type in D has a default initializer.

This is not true any longer. There is disable this().
Jan 18 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 1:29 PM, Walter Bright wrote:
 On 1/17/2014 1:10 PM, H. S. Teoh wrote:
 On Fri, Jan 17, 2014 at 11:43:59AM -0800, Walter Bright wrote:
 [...]
 I've almost never had a problem tracking down the cause of a null
 pointer. Usually just a few minutes with a debugger and getting a
 backtrace.

I think this depends on the kind of code you write. In callback-heavy code, usually when you're multiplexing between many simultaneous request/response chains, these kinds of problems are very hard to track down. You'll see the null pointer somewhere in your callback's context structure, but no amount of backtrace will help you go any further because they all end at the event dispatch loop pretty shortly up the stack, which doesn't tell you where in the chain of events the null came from.

What you do then is go back as far as practical, then put asserts in.

That hardly helps. The source of the problem is not detecting that it occurs, it's avoiding it in the first place by initializing data appropriately. Andrei
Jan 17 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 12:35 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 17 January 2014 at 19:43:58 UTC, Walter Bright wrote:
 I've almost never had a problem tracking down the cause of a null pointer.
 Usually just a few minutes with a debugger and getting a backtrace.

Doesn't work if the unexpected "null" sits in a graph and the source of it is hard to pinpoint or occurs "randomly". E.g. if you are using a "black box" framework or it happens spuriously on a server because it is triggered by a database timeout which never happens on the dev server.

As I replied elsewhere, tracking down the source of a bad value in any variable is a standard debugging problem. There isn't anything special about null in this regard.
Jan 17 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 1:31 PM, Walter Bright wrote:
 On 1/17/2014 12:35 PM, "Ola Fosheim Grøstad"
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 17 January 2014 at 19:43:58 UTC, Walter Bright wrote:
 I've almost never had a problem tracking down the cause of a null
 pointer.
 Usually just a few minutes with a debugger and getting a backtrace.

Doesn't work if the unexpected "null" sits in a graph and the source of it is hard to pinpoint or occurs "randomly". E.g. if you are using a "black box" framework or it happens spuriously on a server because it is triggered by a database timeout which never happens on the dev server.

As I replied elsewhere, tracking down the source of a bad value in any variable is a standard debugging problem. There isn't anything special about null in this regard.

One problem with null is it's not "proportional response", i.e. it takes the application in the back and shoots it in the head instead of e.g. a stale display. That _does_ make null special; there's a large category of application for which aborting is simply not an option. Andrei
Jan 17 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 4:15 PM, Andrei Alexandrescu wrote:
 One problem with null is it's not "proportional response", i.e. it takes the
 application in the back and shoots it in the head instead of e.g. a stale
 display. That _does_ make null special; there's a large category of application
 for which aborting is simply not an option.

I'm aware of that, I was responding to the idea that null pointers are harder to track down once detected. In particular, I don't see how a stale display is an easier bug to track down, although granted it is less annoying to the end user.
Jan 17 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 4:44 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 It isn't special in the theoretical sense, but since it in practice is used for
 a wide variety of things  it becomes a sensitive issue in situations where you
 cannot replay input. Especially since it is used for linking together
 subsystems. In multi user games you can do fine with wrong "float values",
 objects might drift, you might get odd effects, but those bugs can be cool.
They
 might even become features. A wrong "null" makes the client or server shut
down,
 it is disruptive to the service.

 So what can you do? You can log all input events in a ring buffer and send a
big
 dump to the developers when the game client crash and try to find correlation
 between stack trace and input events (if you get multiple reports).  And on a
 server with say 1000 concurrent users that interact and trigger a variety of
 bugs in one big multi-threaded global state world sim… You want to use a safe
 language for world content, where you disable features if needed, but keep the
 world running.

 The same goes for a web service. You don't want to shut down the whole service
 just because the "about page" handler fails to test for null.

 Big systems have to live with bugs, it is inevitable that they run with bugs.
 Erlang was created for stuff like that. Can you imagine a telephone central
 going down just because the settings for one subscriber was wrong and triggered
 a bug?

First off, in all these scenarios you describe, how does not having a null make it EASIER to track down the bug? Next, it's one thing to have a game degrade but keep playing if there's a bug in the code. It's quite another to have a critical system start behaving erratically because of a bug. Remember, a bug means the program has entered an undefined, unanticipated state. If it is a critical system, continuing to run that system is absolutely, positively, the wrong thing to do. The correct way is when the bug is detected, that software is IMMEDIATELY SHUT DOWN and the backup is engaged. If you don't have such a backup, you have a very, very badly designed system. I'm not just making that stuff up; it is not my uninformed opinion. This is how, for example, Boeing designs things for airliners. You wouldn't want it any other way. Fukushima and Deepwater Horizon are what happens when you don't have a system with independent backups. Again, granted, this doesn't apply to a game. Nothing bad happens when a game goes awry. But it definitely applies when there's money involved, like transaction processing, or trading software. And it certainly applies when failure means people die.
Jan 17 2014
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-18 01:22:49 +0000, Walter Bright <newshound2 digitalmars.com> said:

 First off, in all these scenarios you describe, how does not having a 
 null make it EASIER to track down the bug?

Implemented well, it makes it a compilation error. It works like this: - can't pass a likely-null value to a function that wants a not-null argument. - can't assign a likely-null value to a not-null variable. - can't dereference a likely-null value. You have to check for null first, and the check changes the value from likely-null to not-null in the branch taken when the pointer is valid. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 17 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:18 PM, Michel Fortin wrote:
 Implemented well, it makes it a compilation error. It works like this:

 - can't pass a likely-null value to a function that wants a not-null argument.
 - can't assign a likely-null value to a not-null variable.
 - can't dereference a likely-null value.

 You have to check for null first, and the check changes the value from
 likely-null to not-null in the branch taken when the pointer is valid.

I was talking about runtime errors, in that finding the cause of a runtime null failure is not harder than finding the cause of any other runtime invalid value. We all agree that detecting bugs at compile time is better.
Jan 17 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-18 02:41:58 +0000, Walter Bright <newshound2 digitalmars.com> said:

 We all agree that detecting bugs at compile time is better.

I guess so. But you're still unconvinced it's worth it to eliminate null dereferences? -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 17 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 7:23 PM, Michel Fortin wrote:
 I guess so. But you're still unconvinced it's worth it to eliminate null
 dereferences?

I think it's a more nuanced problem than that. But I agree that compile time detect of null reference bugs is better than runtime detection of them.
Jan 17 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 7:38 PM, Walter Bright wrote:
 But I agree that compile time
 detect of null reference bugs is better than runtime detection of them.

BTW, the following program: class C { int a,b; } int test() { C c; return c.b; } When compiled with -O: foo.d(6): Error: null dereference in function _D3foo4testFZi It isn't much, only working on intra-function analysis and only when the optimizer is used, but it's something. It's been in dmd for a long time.
Jan 17 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-01-18 07:10, Walter Bright wrote:

 BTW, the following program:

    class C { int a,b; }

    int test() {
      C c;
      return c.b;
    }

 When compiled with -O:

    foo.d(6): Error: null dereference in function _D3foo4testFZi

 It isn't much, only working on intra-function analysis and only when the
 optimizer is used, but it's something. It's been in dmd for a long time.

Why only when the optimizer is used? -- /Jacob Carlborg
Jan 18 2014
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Jacob Carlborg"  wrote in message news:lbdkvv$2f1n$1 digitalmars.com... 
 Why only when the optimizer is used?

The code that detects the null dereference is in the optimizer.
Jan 18 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-01-18 11:41, Daniel Murphy wrote:

 The code that detects the null dereference is in the optimizer.

Why is it located there? -- /Jacob Carlborg
Jan 18 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Tobias Pankrath"  wrote in message 
news:rxnlyzyfkzndzfslvqzp forum.dlang.org...
 Why is it located there?


Exactly. It _could_ be done in the frontend, but it hasn't.
Jan 18 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-01-18 12:21, Daniel Murphy wrote:

 Exactly.  It _could_ be done in the frontend, but it hasn't.

I see, thanks. -- /Jacob Carlborg
Jan 18 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 2:28 AM, Jacob Carlborg wrote:
 Why only when the optimizer is used?

Yes, because the optimizer uses data flow analysis, and this falls out of that.
Jan 18 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 2:02 AM, Namespace wrote:
 Print nothing, even with -O.

Right. As I explained, it is only intra-function analysis.
Jan 18 2014
prev sibling next sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-18 03:38:31 +0000, Walter Bright <newshound2 digitalmars.com> said:

 On 1/17/2014 7:23 PM, Michel Fortin wrote:
 I guess so. But you're still unconvinced it's worth it to eliminate null
 dereferences?

I think it's a more nuanced problem than that. But I agree that compile time detect of null reference bugs is better than runtime detection of them.

In the C++ project I'm working on (full of asynchronous callbacks) I've built myself a not-null smart pointer type. It tries to disallow things at compile time, but because it can't follow control flow there's obviously some possible leaks that are caught at runtime through an assertion upon assignment. I created it because I had some untraceable bugs and I though it'd be simpler to convert the code base to use not-null pointers than it'd be to narrow the issues manually. The transition was surprisingly easy: just try to make everything you can not-nullable. Sometime the logic forces you to make a variable nullable, sometime you discover a bug in the form of a missing null check. Implementing non-nullable pointers and adapting the code base was worth it for me. I found some bugs that I couldn't track otherwise, and some others that had yet to occur. But more importantly now when I look at a pointer in this project I know immediately from its declaration whether that pointer can be null or not, and I find that very helpful when tweaking code that was written a while ago. But in the process I've created a new meta-language (make_new< T > instead of new, a new smart pointer types, cast function for those pointers, etc.) that someone new on the project will have to learn and train himself to use correctly. Integrating the thing in the language would make things that much simpler because it'd preserve the normal syntax of the language and also because null leaks can all be detected at compile-time, with clearer error messages. This is what motivated my proposal earlier in this thread. I'm proposing an improved version of what I'm already using currently: http://forum.dlang.org/thread/lba1qe$1sih$1 digitalmars.com?page=4#post-lbbgr6:247sf:241:40digitalmars.com I'd like to know what you think of it. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 18 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 7:38 PM, Walter Bright wrote:
 On 1/17/2014 7:23 PM, Michel Fortin wrote:
 I guess so. But you're still unconvinced it's worth it to eliminate null
 dereferences?

I think it's a more nuanced problem than that.

I should clarify that. I oppose solutions that replace the null seg fault with another bug that is harder to detect. (I know this is not what you have proposed.)
Jan 18 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 3:34 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 do. The correct way is when the bug is detected, that software is IMMEDIATELY
 SHUT DOWN and the backup is engaged. If you don't have such a backup, you have
 a very, very badly designed system.

In that case 99.99% of all software is very, very badly designed. DMD inclusive.

You elided the qualification "If it is a critical system". dmd is not a safety critical application.
Jan 18 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 11:11 PM, Kapps wrote:
 On Sunday, 19 January 2014 at 02:33:15 UTC, Walter Bright wrote:
 You elided the qualification "If it is a critical system". dmd is not a safety
 critical application.

Nor are 99.99%, possibly 100%, of the applications currently being built with D.

Sociomantic uses D to write trading software. I think they'd be ill advised to write code in such a way that it continues making trades after entering an invalid state, as in it could be very expensive. Furthermore, part of the reason why I am adamant about this is far too often I run into, including in this thread, programmers who believe that the way to write critical software is to keep the program running even if it has failed.
 The points are valid, no safety critical application should ever rely on any
 individual components not failing. But for the purpose of D, they are not
 particularly applicable.

I don't buy that this is not applicable for D. D must not promote unsafe (as in potentially life threatening) programming practice as a "best practice". You never know how somebody is going to use a programming language.
 People are just arguing over two different things
 (safety critical code vs standard D code), with neither disagreeing with the
 others, simply bringing up different situations.

There has been a lot of misunderstandings and miscommunications in this thread. I do my best to clear them up.
Jan 18 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 6:33 PM, Walter Bright wrote:
 You elided the qualification "If it is a critical system". dmd is not a safety
 critical application.

And I still practice what I preach with DMD. DMD never attempts to continue running after it detects that it has entered an invalid state - it ceases immediately. Furthermore, when it detects any error in the source code being compiled, it does not generate an object file.
Jan 18 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2014 4:20 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 If you take a theoretical position (which I think you do)

Please set such statements against my assertion that Boeing follows the principles I outlined in practice, and very successfully.
Jan 19 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-01-19 00:34, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:

 I have not argued for not having a null. I have argued for trapping
 null, instantiating a type specific default and recover if the type's
 meta says so. That default could be to issue a NullException. At that
 point you should be able to log null dereferences too.

I think nil (null) works quite nicely in Ruby. nil is a singleton instance of the NilClass class. Since it's an object you can call methods on it, like to_s, which returns an empty string. It works quite well when doing web development with Ruby on Rails. If you're trying to render something that's nil you'll get nothing, instead of crashing that whole page. Sure, there might be a small icon or similar that isn't rendered but that's usually a minor detail. If the page is working and main content is rendered that's preferable. -- /Jacob Carlborg
Jan 19 2014
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-19 11:30:00 +0000, "bearophile" <bearophileHUGS lycos.com> said:

 Jacob Carlborg:
 
 I think nil (null) works quite nicely in Ruby. nil is a singleton 
 instance of the NilClass class. Since it's an object you can call 
 methods on it, like to_s, which returns an empty string. It works quite 
 well when doing web development with Ruby on Rails. If you're trying to 
 render something that's nil you'll get nothing, instead of crashing 
 that whole page. Sure, there might be a small icon or similar that 
 isn't rendered but that's usually a minor detail. If the page is 
 working and main content is rendered that's preferable.

Walter is arguing against this solution in D. While that can be OK for Ruby used for web development, for a statically typed language meant for safe coding styles, there are better type-based solution to that problem.

It won't work in D's type system anyway. Ruby is dynamically typed, that's why it can work. Interestingly, in Objective-C calling a method on a null object pointer just does nothing. That's a feature that is often useful when chaining calls that might return null (you don't have to do all these extra checks) or with weak pointers, but it can on occasion lead to subtle bugs. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 19 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 4:44 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 Big systems have to live with bugs, it is inevitable that they run with bugs.

It's a dark and stormy night. You're in a 747 on final approach, flying on autopilot. Scenario 1 ---------- The autopilot software was designed by someone who thought it should keep operating even if it detects faults in the software. The software runs into a null object when there shouldn't be one, and starts feeding bad values to the controls. The plane flips over and crashes, everybody dies. But hey, the software kept on truckin'! Scenario 2 ---------- The autopilot software was designed by Boeing. Actually, there are two autopilots, each independently developed, with different CPUs, different hardware, different algorithms, different languages, etc. One has a null pointer fault. A deadman circuit sees this, and automatically shuts that autopilot down. The other autopilot immediately takes over. The pilot is informed that one of the autopilots failed, and the pilot immediately shuts off the remaining autopilot and lands manually. The passengers all get to go home. Note that in both scenarios there are bugs in the software. Yes there have been incidents with earlier autopilots where bugs in it caused the airplane to go inverted. Consider also the Toyota. My understanding from reading reports (admittedly journalists botch up the facts) is that a single computer controls the brakes, engine, throttle, ignition switch, etc. Oh joy. I wouldn't want to be in that car when it keeps on going despite having self-detected faults. It could, you know, start going at full throttle and ignore all signals to brake or turn off, only stopping when it crashes or runs out of gas.
Jan 17 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:22 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Saturday, 18 January 2014 at 01:46:55 UTC, Walter Bright wrote:
 The autopilot software was designed by someone who thought it should keep
 operating even if it detects faults in the software.

I would not write autopilot or life-support software in D. So that is kind of out-of-scope for the language. But: Keep the system simple, select a high level language and verify correctness by an automated proof system. Use 3 independently implemented systems and shut down the one that produces deviant values. That covers more ground than the unlikely null-pointers in critical systems. No need to self-detect anything.

I didn't mention that the dual autopilots also have a comparator on the output, and if they disagree they are both shut down. The deadman is an additional check. The dual system has proven itself, a third is not needed.
 Consider also the Toyota. My understanding from reading reports (admittedly
 journalists botch up the facts) is that a single computer controls the brakes,
 engine, throttle, ignition switch, etc. Oh joy. I wouldn't want to be in that
 car when it keeps on going despite having self-detected faults.

So you would rather have the car drive off the road because the anti-skid software abruptly turned itself off during an emergency manoeuvre?

Please reread what I wrote. I said it shuts itself off and engages the backup, and if there is no backup, you have failed at designing a safe system.
 But would you stay in a car where the driver talks in a cell-phone while
 driving, or would you tell him to stop? Probably much more dangerous if you
 measured correlation between accidents and system features. So you demand
 perfection from a computer, but not from a human being that is exhibiting
 risk-like behaviour. That's an emotional assessment.

 The rational action would be to improve the overall safety of the system,
rather
 than optimizing a single part. So spend the money on installing a cell-phone
 jammer and an accelerator limiter rather than investing in more computers.
 Clearly, the computer is not the weakest link, the driver is. He might not
 agree, but he is and he should be forced to exhibit low risk behaviour. Direct
 effort to where it has most effect.

 (From a system analytical point of view. It might not be a good sales tactic,
 because car buyers aren't that rational.)

I have experience with this stuff, Ola, from my years at Boeing designing flight critical systems. What I outlined is neither irrational nor emotionally driven, and has the safety record to prove its effectiveness. I also ask that you please reread what I wrote - I explicitly do not demand perfection from a computer.
Jan 17 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:40 PM, deadalnix wrote:
 Different software, different needs. Ultimately, that distinction
 is irrelevant anyway. The whole possibility of these scenarios
 can be avoided in case of null dereferences by proper language
 design.

I've already agreed that detecting bugs at compile time is better.
Jan 17 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 7:05 PM, H. S. Teoh wrote:
 [...]

Thank you, a good explanation. I don't know how the anti-skid brake system is designed. But on the older systems, brakes are a dual mostly-independent system. There was one system for the front brakes, and another for the rear. Dual cylinders, dual reservoirs, etc. The brake pedal operated both cylinders. There was even a hydraulic comparator between the two, which would turn on a red [brake] light on the dash if they differed in pressure. Last year, I acquired a leak in my rear brakes on my old truck, and that light coming on was my first indication of trouble. The front brakes still worked fine, and topping off the rear reservoir got the rear brakes temporarily working, and I was able to ease it to the repair shop without difficulty. It's a good example of how to build a safe, fault tolerant system.
Jan 17 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 11:43 AM, Walter Bright wrote:
 On 1/17/2014 6:06 AM, "Ola Fosheim Grøstad"
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 17 January 2014 at 01:42:38 UTC, Andrei Alexandrescu wrote:
 One common idiom to replace null pointer exceptions with milder
 reproducible
 errors is the null object pattern, i.e. there is

Usually null failures are hard to track when some function returned a null value as an indication of error when the programmer expected an exception.

I've almost never had a problem tracking down the cause of a null pointer. Usually just a few minutes with a debugger and getting a backtrace.

I think this is bias that is countered by extensive experience of groups at Facebook that have no axe to grind in the matter. This bias makes it all the more difficult to recognize the importance of the problem. Andrei
Jan 17 2014
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 17, 2014 at 07:11:16PM -0800, Walter Bright wrote:
[...]
 We've talked about the floating point issue, where nan values
 "poison" the results rather than raise a seg fault. I remembered
 later that a while back Don tried to change D to throw an exception
 when a floating point nan was encountered, specifically because this
 made it easier for him to track down the source of the nan. He said
 it was pretty hard to backtrack it from the eventual output.

This is tangential to the discussion, but the nan issue got me thinking about how one might detect the source of a nan. One crude way might be to use a pass-thru function that asserts if its argument is a nan: real assumeNotNan(real v, string file=__FILE__, size_t line=__LINE__) { assert(!isNan(v), "assumeNotNan failed at " ~ file ~ " line " ~ to!string(line)); return v; } Then you can stick this inside a floating-point expression in the same way you'd stick assert(ptr !is null) in some problematic code in order to find where the null is coming from, to narrow down the source of the nan: real complicatedComputation(real x, real y, real z) { // Original code: //return sqrt(x) * y - z/(x-y) + suspiciousFunc(x,y,z); // Instrumented code (UFCS rawkz!): return sqrt(x).assumeNotNan * y - (z/(x-y)).assumeNotNan + suspiciousFunc(x,y,z).assumeNotNan; } Makes it *slightly* easier than having to break up a complex expression and introduce temporaries in order to insert asserts between terms. But, granted, not by much. Still, it reduces the pain somewhat. T -- Political correctness: socially-sanctioned hypocrisy.
Jan 17 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 17 January 2014 at 14:06:59 UTC, Ola Fosheim Grøstad 
wrote:
 Like if you delete a User object, then all "weak" pointers are 
 set to a UnknownUser default object. So that you can call 
 user.isLoggedIn() and get false, but you should be able to 
 default to a different Class to get a different set of virtual 
 calls.

This particular thing is easy enough to do with null too: bool isLoggedIn(User user) { return user is null ? false : user.isLoggedInImpl(); }
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 17 January 2014 at 14:13:40 UTC, Adam D. Ruppe wrote:
 bool isLoggedIn(User user) {
     return user is null ? false : user.isLoggedInImpl();
 }

In most cases you can avoid null-related issues if you plan for it, but if you adapt existing software you might not afford to do the changes. Even simple adaption of a framework can cost many weeks of hard work and then you need to keep it in sync with future versions of the framework. I also think there might be cases where you run the null-test a lot of times with the same result and would be better without it. Like in deep nested data structures where the pointer almost never is null, and then a trap with recovery might be cheaper, amortized. Sometimes you can write more efficient datastructures if you ban deletion: Like: ptr = someobject; weakcontainer.add(ptr); ... ptr.setdeleted(); ptr = null; ... weakptr = weakcontainer.get(id); if(weakptr.isValid())...; In this case weakptr would point to the original object before GC, and to the init object after GC. Saving space, but not requiring changes to the weakcontainer implementation.
Jan 17 2014
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-17 01:42:37 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Further avenues are opened by thinking what happens if e.g. init is 
 private or  disable-d.
 
 Thoughts?

Some more thoughts. Couldn't we just add a class declaration attribute that'd say that by default this type is not-null? For instance: // references to A are not-null by default notnull class A {} // references to B are not-null too (inherited) class B : A {} Then you have two types: A // not-null reference to A. A? // nullable reference to A. Use it like this: A a; // error: cannot be null, default initialization not allowed A? a; // default-initialized to null void test1(A a) // param a cannot be null { a.foo(); // fine } void test2(A? a) // param a can be null { a.foo(); // error: param a could be null if (a) a.foo(); // param a cannot be null, allowed } void test3(A? a) // param a can be null { A aa = a; // error: param a could be null, can't init to not-null type if (a) aa = a; // param a cannot be null, allowed } This is basically what everyone would wish for a not-null type to do. The syntax is clean and the control flow forces you to check for null before use. Misuses result in a compile-time error. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 17 2014
next sibling parent reply "Yota" <yotaxp thatGoogleMailThing.com> writes:
On Friday, 17 January 2014 at 15:05:10 UTC, Michel Fortin wrote:
 On 2014-01-17 01:42:37 +0000, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:

 Further avenues are opened by thinking what happens if e.g. 
 init is private or  disable-d.
 
 Thoughts?

Some more thoughts. Couldn't we just add a class declaration attribute that'd say that by default this type is not-null?

I have been hoping D would become not-null-by-default since I discovered the language. Looking at what it has done with other type classes, (shared, immutable, etc) it always seemed like something that would be right up its alley. However in my dream world, it would simply apply to all reference types by default. I'm not a fan of using an attribute for this due to how hidden it is from the programmer. You should be able to determine the nullability of something from looking at the code that uses it. But then again, I feel like there should also be a way to distinguish value types from reference types just by looking at the usage, so what do I know? =p
Jan 17 2014
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-17 17:52:18 +0000, "Yota" <yotaxp thatGoogleMailThing.com> said:

 On Friday, 17 January 2014 at 15:05:10 UTC, Michel Fortin wrote:
 Couldn't we just add a class declaration attribute that'd say that by 
 default this type is not-null?

I have been hoping D would become not-null-by-default since I discovered the language. Looking at what it has done with other type classes, (shared, immutable, etc) it always seemed like something that would be right up its alley. However in my dream world, it would simply apply to all reference types by default.

Well, nothing prevents that from happening if notnull is adopted. As proposed, notnull is designed to keep the language backward compatible: it's opt-in and you don't even have to opt-in all at once. Potential problems the early implementation might have can only burn those who give it a try. One can hope the feature will become so loved that it'll become standard pointer behaviour one day, but that can't happen as long as the feature is theoretical. In the eventuality we decide one day to push not-null behaviour to the rest of the language, notnull will just become an attribute with no effect. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 18 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-01-17 16:05, Michel Fortin wrote:

 This is basically what everyone would wish for a not-null type to do.
 The syntax is clean and the control flow forces you to check for null
 before use. Misuses result in a compile-time error.

Yes, exactly. But the issue is that the compiler needs to be modified for this. -- /Jacob Carlborg
Jan 17 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-17 20:21:36 +0000, Jacob Carlborg <doob me.com> said:

 On 2014-01-17 16:05, Michel Fortin wrote:
 
 This is basically what everyone would wish for a not-null type to do.
 The syntax is clean and the control flow forces you to check for null
 before use. Misuses result in a compile-time error.

Yes, exactly. But the issue is that the compiler needs to be modified for this.

Andrei's post was referring at language/compiler changes too: allowing init to be defined per-class, with a hint about disabling init. I took the hint that modifying the compiler to add support for non-null was in the cards and proposed something more useful and less clunky to use. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 17 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 12:42 PM, Michel Fortin wrote:
 On 2014-01-17 20:21:36 +0000, Jacob Carlborg <doob me.com> said:

 On 2014-01-17 16:05, Michel Fortin wrote:

 This is basically what everyone would wish for a not-null type to do.
 The syntax is clean and the control flow forces you to check for null
 before use. Misuses result in a compile-time error.

Yes, exactly. But the issue is that the compiler needs to be modified for this.

Andrei's post was referring at language/compiler changes too: allowing init to be defined per-class, with a hint about disabling init. I took the hint that modifying the compiler to add support for non-null was in the cards and proposed something more useful and less clunky to use.

Yes, improving the language is in the cards. I have collected enough hard evidence to convince myself that null references are a real and important issue (previously I'd agreed with Walter who is considering it not particularly remarkable). Andrei
Jan 17 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 4:21 PM, Adam D. Ruppe wrote:
 On Saturday, 18 January 2014 at 00:12:16 UTC, Andrei Alexandrescu wrote:
 Yes, improving the language is in the cards. I have collected enough
 hard evidence to convince myself that null references are a real and
 important issue (previously I'd agreed with Walter who is considering
 it not particularly remarkable).

Have they tried using a NotNull!T yet?

Not a D project (Java/Android). However there is work underway on a static analyzer for Java. I cannot give exact stats, but we have gathered extensive data that null pointer exceptions form a major, major problem on Facebook's Android app, and that virtually all can be solved by static analysis. No amount of speculation and hypothesizing will talk me out of that. Andrei
Jan 17 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 4:25 PM, Andrei Alexandrescu wrote:
 On 1/17/14 4:21 PM, Adam D. Ruppe wrote:
 On Saturday, 18 January 2014 at 00:12:16 UTC, Andrei Alexandrescu wrote:
 Yes, improving the language is in the cards. I have collected enough
 hard evidence to convince myself that null references are a real and
 important issue (previously I'd agreed with Walter who is considering
 it not particularly remarkable).

Have they tried using a NotNull!T yet?

Not a D project (Java/Android). However there is work underway on a static analyzer for Java. I cannot give exact stats, but we have gathered extensive data that null pointer exceptions form a major, major problem on Facebook's Android app, and that virtually all can be solved by static analysis. No amount of speculation and hypothesizing will talk me out of that.

I don't disagree that some form of static analysis that can detect null dereferencing at compile time (i.e. static analysis) is a good thing. I concur that detecting bugs at compile time is better. I don't disagree that for an app like a game, soldiering on after the program is in an invalid state doesn't do any particular harm, and so one can make a good case for using the null object pattern. I do not agree with the notion that null pointers detected at runtime are harder to track down than other kinds of invalid values detected at runtime. I strong, strongly, disagree with the notion that critical systems should soldier on once they have entered an invalid state. Such is absolutely the wrong way to go about making a fault tolerant system. For hard evidence, I submit the safety record of airliners.
Jan 17 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:12 PM, bearophile wrote:
 Walter Bright:

 I strong, strongly, disagree with the notion that critical systems should
 soldier on once they have entered an invalid state.

The idea is to design the language and its type system (and static analysis tools) to be able to reduce the frequency (or probability) of such invalid states, because many of them are removed while you write the program.

Once again, "I don't disagree that some form of static analysis that can detect null dereferencing at compile time (i.e. static analysis) is a good thing. I concur that detecting bugs at compile time is better."
Jan 17 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/17/14 5:58 PM, Walter Bright wrote:
 I strong, strongly, disagree with the notion that critical systems
 should soldier on once they have entered an invalid state. Such is
 absolutely the wrong way to go about making a fault tolerant system. For
 hard evidence, I submit the safety record of airliners.

You're arguing against a strawman, and it is honestly not pleasant that you reduced one argument to the other. Andrei
Jan 17 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:56 PM, Andrei Alexandrescu wrote:
 On 1/17/14 5:58 PM, Walter Bright wrote:
 I strong, strongly, disagree with the notion that critical systems
 should soldier on once they have entered an invalid state. Such is
 absolutely the wrong way to go about making a fault tolerant system. For
 hard evidence, I submit the safety record of airliners.

You're arguing against a strawman, and it is honestly not pleasant that you reduced one argument to the other.

I believe I have correctly separated the various null issues into 4 separate ones, and have tried very hard not to conflate them.
Jan 17 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 6:42 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 But then you have to define "invalid state",

An unexpected value is an invalid state.
Jan 17 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/19/2014 12:01 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Saturday, 18 January 2014 at 02:59:43 UTC, Walter Bright wrote:
 On 1/17/2014 6:42 PM, "Ola Fosheim Grøstad"
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 But then you have to define "invalid state",

An unexpected value is an invalid state.

It is only an invalid state for a subsystem, if your code is written to handle it, it can contain it and recover (or disable that subsystem). Assuming that you know that it unlikely to be caused by memory corruption. ...

This is not a plausible assumption. What you tend to know is that the program is unlikely to fail because otherwise it would not have been shipped, being safety critical. I.e. when it fails, you don't know that it is unlikely to be caused by something. It could be hardware failure, and even a formal correctness proof does not help with that.
 The problem with being rigid on this definition is ...

He is not.
 What is the essential difference between insisting on stopping a program
 with bugs and insisting on not starting a program with bugs? There is no
 difference.
 ...

Irrelevant. He is arguing for stopping the system once it has _become clear_ that the _current execution_ might not deliver the expected results.
Jan 19 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/20/2014 6:18 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 Think about the effect of this: 1 router detects a bug, by the logic in this
 thread it should then notify all routers running the same software and tell
them
 to shut down immediately. Result: insta-death to entire Internet.

Not only has nobody suggested this, I have explicitly written here otherwise, more than once. I infer you think that my arguments here are based on something I dreamed up in 5 minutes of tip-tapping at my keyboard. They are not. They are what Boeing and the aviation industry use extremely successfully to create incredibly safe airliners, and the track record is there for anyone to see. It's fine if believe you've found a better way. But there's a high bar of existing practice and experience to overcome with a better way, and a need to start from a position of understanding that successful existing practice first.
Jan 20 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-17 21:42, Michel Fortin wrote:

 Andrei's post was referring at language/compiler changes too: allowing
 init to be defined per-class, with a hint about disabling init. I took
 the hint that modifying the compiler to add support for non-null was in
 the cards and proposed something more useful and less clunky to use.

Hehe, right, I kind of lost track of the discussion. -- /Jacob Carlborg
Jan 18 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 18 January 2014 at 00:12:16 UTC, Andrei Alexandrescu 
wrote:
 Yes, improving the language is in the cards. I have collected 
 enough hard evidence to convince myself that null references 
 are a real and important issue (previously I'd agreed with 
 Walter who is considering it not particularly remarkable).

Have they tried using a NotNull!T yet? I wrote one ages ago for a phobos pull request that didn't really go anywhere, but I'm still keeping my file updated with new ideas (most recently, the if(auto nn = x.checkNull) {} thing): http://arsdnet.net/dcode/notnull.d I really think that is very close to meeting all the requirements here, but tbh I don't use it much in the real world myself so I could be missing a major flaw. Of course, such library solutions don't change the default nullability of references.
Jan 17 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 18 January 2014 at 00:25:49 UTC, Andrei Alexandrescu
wrote:
 On 1/17/14 4:21 PM, Adam D. Ruppe wrote:
 On Saturday, 18 January 2014 at 00:12:16 UTC, Andrei 
 Alexandrescu wrote:
 Yes, improving the language is in the cards. I have collected 
 enough
 hard evidence to convince myself that null references are a 
 real and
 important issue (previously I'd agreed with Walter who is 
 considering
 it not particularly remarkable).

Have they tried using a NotNull!T yet?

Not a D project (Java/Android). However there is work underway on a static analyzer for Java. I cannot give exact stats, but we have gathered extensive data that null pointer exceptions form a major, major problem on Facebook's Android app, and that virtually all can be solved by static analysis. No amount of speculation and hypothesizing will talk me out of that. Andrei

I remember mentioning that a year or two month ago, and you dismissing it. Happy to see you changed your mind on that.
Jan 17 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 I strong, strongly, disagree with the notion that critical 
 systems should soldier on once they have entered an invalid 
 state.

The idea is to design the language and its type system (and static analysis tools) to be able to reduce the frequency (or probability) of such invalid states, because many of them are removed while you write the program. Bye, bearophile
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 18 January 2014 at 01:58:13 UTC, Walter Bright wrote:
 I strong, strongly, disagree with the notion that critical 
 systems should soldier on once they have entered an invalid 
 state. Such is absolutely the wrong way to go about making a 
 fault tolerant system. For hard evidence, I submit the safety 
 record of airliners.

But then you have to define "invalid state", not in terms of language constructs, but in terms of the model the implementation is based on. If your thread only uses thread local memory and safe language features it should be sufficient to spin down that thread and restart that sub-service. That's what fault tolerant operating systems do. In a functional language it is easy, you may keep computing with "bottom" until it disappears. "bottom" OR true => true, "bottom" AND false => false. You might even succeed by having lazy evaluation over functions that would never halt. If you KNOW that a particular type is not going to have any adverse affect if taking a default object rather than null (could probably even prove it in is some cases), it does not produce an "invalid state". It might be an exceptional state, but that does not imply that it is invalid. Some systems are in an pragmatic fuzzy state. Take fuzzy logic as an example (http://www.dmitry-kazakov.de/ada/fuzzy.htm) where you operate with two fluid dimensions: necessity and possibility, representing a space with the extremes: false, true, contradiction and uncertain. There is no "invalid state" if the system is designed to be in a state of "best effort".
Jan 17 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/17/2014 7:05 AM, Michel Fortin wrote:
 Some more thoughts.

The postfix ? has parsing issues with ?: expressions. Aside from that, non-null is only one case of a universe of types that are subsets of other types. I'd prefer a more general solution.
Jan 18 2014
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-18 21:05:01 +0000, Walter Bright <newshound2 digitalmars.com> said:

 On 1/17/2014 7:05 AM, Michel Fortin wrote:
 Some more thoughts.

The postfix ? has parsing issues with ?: expressions.

I thought that because the left hand side was a type it could be disambiguated. Somewhat like '*' is disambiguated between pointer declaration and multiplication depending on whether the left hand side is a type. int * a; // pointer declaration a * b; // multiplication But I might be wrong, especially since ?: is a trinary operator while * is always binary.
 Aside from that, non-null is only one case of a universe of types that 
 are subsets of other types. I'd prefer a more general solution.

Hum, kind of thing do we want to support in a general solution? The closest thing I can think of is range constrains. Here's an example (invented syntax): void test(int!0..10 a) // param a is in 0..10 range { int b = a; // fine int!0..5 c = a; // error, incompatible ranges int!0..5 d = a/2; // fine, can't escape range if (a < 5) d = a; // fine, 'a' is int$0..5 on this branch } Obviously compiler code for range constrains and non-null pointers would be shared to some extent, as the rules are pretty similar. But what other cases should be covered by a more general solution? Do we want to add a way for users to declare their own custom type modifiers that can do things? That depends on control flow? -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 18 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 1:38 PM, Michel Fortin wrote:
 The closest thing I can think of is range constrains. Here's an example
 (invented syntax):

I don't think a new syntax is required. We already have the template syntax: RangedInt!(0,10) should do it.
Jan 18 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-18 21:57:21 +0000, Walter Bright <newshound2 digitalmars.com> said:

 On 1/18/2014 1:38 PM, Michel Fortin wrote:
 The closest thing I can think of is range constrains. Here's an example
 (invented syntax):

I don't think a new syntax is required. We already have the template syntax: RangedInt!(0,10) should do it.

It works, up to a point. void foo(RangedInt!(0, 5) a); void bar(RangedInt!(0, 10) a) { if (a < 5) foo(a); // what happens here? } In that "foo(a)" line, depending on the implementation of RangedInt you either get a compile-time error that RangedInt!(0, 10) can't be implicitly converted to RangedInt!(0, 5) and have to explicitly convert it, or you get implicit conversion with a runtime check that throws. Just like pointers, not knowing about the actual control flow pushes range constrains enforcement at runtime in situations like this one. It's better than nothing since it'll throw immediately when passing an out of range value to a function and thus the wrong value won't propagate further, but static analysis would make this much better. In fact, even the most obvious case can't be caught at compile-time with the template approach: void baz() { foo(999); // runtime error here } -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 18 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 2:16 PM, Michel Fortin wrote:
 It works, up to a point.

      void foo(RangedInt!(0, 5) a);

      void bar(RangedInt!(0, 10) a)
      {
          if (a < 5)
              foo(a); // what happens here?
      }

 In that "foo(a)" line, depending on the implementation of RangedInt you either
 get a compile-time error that RangedInt!(0, 10) can't be implicitly converted
to
 RangedInt!(0, 5) and have to explicitly convert it, or you get implicit
 conversion with a runtime check that throws.

Yes, and I'm not seeing the problem. (The runtime check may also be removed by the optimizer.)
 Just like pointers, not knowing about the actual control flow pushes range
 constrains enforcement at runtime in situations like this one.

With pointers, the enforcement only happens when converting a pointer to a nonnull pointer.
 In fact, even the most obvious case can't be caught at compile-time with the
 template approach:

      void baz()
      {
          foo(999); // runtime error here
      }

Sure it can. Inlining, etc., and appropriate use of compile time constraints.
Jan 18 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
The point being, there is a whole universe of subset types. You cannot begin to 
predict all the use cases, let alone come up with a special syntax for each.
Jan 18 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 2:43 PM, bearophile wrote:
 Walter Bright:

 The point being, there is a whole universe of subset types. You cannot begin
 to predict all the use cases, let alone come up with a special syntax for each.

On the other hand there are some groups of types that are both very common, lead to a good percentage of bugs and troubles, and need a refined type semantic & management to be be handled well (so it's hard to implement them as library types).

That would be a problem with D's ability to define library types, and we should address that rather than take the Go approach and add more magic to the compiler.
 Currently you can't implement
 "good enough" not-nullable reference types or ranged integers in D).

This is not at all clear.
 4) Give up and accept to use a simpler language, with a simpler compiler, that
 is more easy to create and develop.

Applications are complicated. A simple language tends to push the complexity into the source code. Java is a fine example of that.
Jan 18 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2014 3:10 PM, bearophile wrote:
 Walter Bright:

 Currently you can't implement
 "good enough" not-nullable reference types or ranged integers in D).

This is not at all clear.

A good Ranged should allow syntax like this, and it should catch this error at compile time (with an "enum precondition"): Ranged!(int, 0, 10)[] arr = [1, 5, 12, 3, 2]; It also should use the CPU overflow/carry flags to detect efficiently enough integer overflows on a Ranged!(uint, 0, uint.max) type. It should handle the conversions nicely to the super-type and allow the usage of a ranged int as array index. And array bound tests should be disabled if you are using a ranged size_t that is statically known to be in the interval of the array, because this is one of the main purposes of ranged integrals. And D arrays should have optional strongly-typed index types, as in Ada. Because this makes the code safer, easier to reason about, and even faster (thanks to disabling some now unnecessary array bound tests).

While these are all desirable features, it is not clear that these cannot be implemented without changing the language - for example, improved optimization can do a lot. And secondly, you said "good enough" not "perfect". Even putting things into the language does not imply they will be perfect, as experience amply shows.
 Similarly not-nullable pointers and class references have some semantic
 requirements that are not easy to implement in D today.

For example?
Jan 18 2014
prev sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-18 22:28:14 +0000, Walter Bright <newshound2 digitalmars.com> said:

 On 1/18/2014 2:16 PM, Michel Fortin wrote:
 It works, up to a point.
 
      void foo(RangedInt!(0, 5) a);
 
      void bar(RangedInt!(0, 10) a)
      {
          if (a < 5)
              foo(a); // what happens here?
      }
 
 In that "foo(a)" line, depending on the implementation of RangedInt you either
 get a compile-time error that RangedInt!(0, 10) can't be implicitly 
 converted to
 RangedInt!(0, 5) and have to explicitly convert it, or you get implicit
 conversion with a runtime check that throws.

Yes, and I'm not seeing the problem. (The runtime check may also be removed by the optimizer.)

I'm not concerned about performance, but about catching bugs early. You said it: the compiler (the optimizer) knows (more or less) whether it is possible for you to have an error here (which is why the check might disappear as dead code), but it will let it pass and make the generated code wait until a bad value is passed at runtime to throw something. Couldn't we tell the compiler to emit an error if a function argument "might" be in the offending range instead? That'd be much more useful to find bugs, because you'd find them at compile-time.
 Just like pointers, not knowing about the actual control flow pushes range
 constrains enforcement at runtime in situations like this one.

With pointers, the enforcement only happens when converting a pointer to a nonnull pointer.

True.
 In fact, even the most obvious case can't be caught at compile-time with the
 template approach:
 
      void baz()
      {
          foo(999); // runtime error here
      }

Sure it can. Inlining, etc., and appropriate use of compile time constraints.

Inlining will not allow the error to be caught at compile time, although it might allow the runtime check to be eliminated as dead code. But this is not about performance, it's about detecting this kind of bug early (at compile time). -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 18 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/18/2014 10:05 PM, Walter Bright wrote:
 On 1/17/2014 7:05 AM, Michel Fortin wrote:
 Some more thoughts.

The postfix ? has parsing issues with ?: expressions. ...

In what sense? It can be unambiguously parsed easily.
 Aside from that, non-null is only one case of a universe of types that
 are subsets of other types.

This is not true. The main rationale for "non-null" is to eliminate null dereferences. A? in his proposal is different from current nullable references in that the compiler does not allow them to be dereferenced. If we just had a solution for arbitrary subset types, we'd _still_ be left with a type of references that might be null, but are not prevented to be dereferenced. Besides, I think it is strange to think of valid references as just being some arbitrary special case of nullable references. A nullable reference is roughly what you get when you put an arbitrary incompatible value 'null' into the set of valid references: {valid references} ∪ {null} All that is asked for is to make the disjunct that is _actually interesting_ it's own type. Sure, it could be described as: {x ∈ ({valid references} ∪ {null}) | x ≠ null} But I think this is a silly way of expressing oneself.
 I'd prefer a more general solution.

Subset types are not more general than the proposed feature.
Jan 18 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-19 07:56:06 +0000, Timon Gehr <timon.gehr gmx.ch> said:

 On 01/18/2014 10:05 PM, Walter Bright wrote:
 Aside from that, non-null is only one case of a universe of types that
 are subsets of other types.

This is not true. The main rationale for "non-null" is to eliminate null dereferences. A? in his proposal is different from current nullable references in that the compiler does not allow them to be dereferenced.

Actually, 'A?' would implicitly convert to 'A' where the compiler can prove control flow prevents its value from being null. So you can dereference it in a branch that checked for null: class A { int i; void foo(); } void bar(A a); // non-nullable parameter void test(A? a, A? a2) { a.i++; // error, 'a' might be null a.foo(); // error, 'a' might be null bar(a); // error, 'a' might be null if (a) { a.i++; // valid, 'a' can't be null here a.foo(); // valid, 'a' can't be null here bar(a); // valid, 'a' can't be null here } } Obviously, the compiler has to be pessimistic, which means that if your control flow is too complicated you might have to use a cast, or add an extra "if" or assert. Personally, I don't see that as a problem. If I have to choose between dynamic typing and static typing, I'll choose the later, even if some time the type system forces me to do a cast. Same thing here with null.
 If we just had a solution for arbitrary subset types, we'd _still_ be 
 left with a type of references that might be null, but are not 
 prevented to be dereferenced.

And I left out that point while converting the example to numeric ranges earlier. It's an important point. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 19 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/19/2014 01:03 PM, Michel Fortin wrote:
 Actually, 'A?' would implicitly convert to 'A' where the compiler can
 prove control flow prevents its value from being null.

I think the type should be upgraded. i.e.:
 So you can
 dereference it in a branch that checked for null:

      class A { int i; void foo(); }
      void bar(A a); // non-nullable parameter

      void test(A? a, A? a2)
      {
          a.i++; // error, 'a' might be null
          a.foo(); // error, 'a' might be null
          bar(a); // error, 'a' might be null

          if (a)
          {

              a.i++; // valid [...]
              a.foo(); // valid [...]
              bar(a); // valid [...]
          }
      }

Jan 19 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-01-19 20:07:40 +0000, Timon Gehr <timon.gehr gmx.ch> said:

 On 01/19/2014 01:03 PM, Michel Fortin wrote:
 Actually, 'A?' would implicitly convert to 'A' where the compiler can
 prove control flow prevents its value from being null.

I think the type should be upgraded. i.e.:
 So you can
 dereference it in a branch that checked for null:
 
      class A { int i; void foo(); }
      void bar(A a); // non-nullable parameter
 
      void test(A? a, A? a2)
      {
          a.i++; // error, 'a' might be null
          a.foo(); // error, 'a' might be null
          bar(a); // error, 'a' might be null
 
          if (a)
          {

              a.i++; // valid [...]
              a.foo(); // valid [...]
              bar(a); // valid [...]
          }
      }


That's one way to do it. Note that this means you can't assign null to 'a' inside the 'if' branch. But I wouldn't worry too much about that. I think it'd make a good first implementation. What I expect from a not-null feature is that it starts by being over-restrictive and with time, as the control flow analysis evolves, unnecessary restrictions would be lifted. That's similar to how CTFE and purity became what they are today. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Jan 19 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01/20/2014 01:44 AM, Michel Fortin wrote:
 That's one way to do it. Note that this means you can't assign null to
 'a' inside the 'if' branch.  ...

Such an assignment could downgrade the type again. An alternative would be to not use flow analysis at all and require eg: A? a=foo(); if(A b=a){ // use b of type 'A' here } A solution that sometimes allows A? to be dereferenced will likely have issues with eg. IFTI.
Jan 20 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Saturday, 18 January 2014 at 21:05:02 UTC, Walter Bright wrote:
 On 1/17/2014 7:05 AM, Michel Fortin wrote:
 Some more thoughts.

The postfix ? has parsing issues with ?: expressions. Aside from that, non-null is only one case of a universe of types that are subsets of other types. I'd prefer a more general solution.

Whats about a new storage class/modifier like safe_ref/ disable_null/ disallow_null? ---- class A { } void test( safe_ref A) { } A a; test(a); // error ---- and maybe also: ---- safe_ref A a; // error ----
Jan 18 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
bearophile proposes the following syntax:
 T? means T nullable
 T  = means not nullable.

Here: http://d.puremagic.com/issues/show_bug.cgi?id=4571 And here: http://forum.dlang.org/thread/mailman.372.1364547485.4724.digitalmars-d puremagic.com?page=6#post-jlqdndoqeuhixxcmcbar:40forum.dlang.org
Jan 18 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Namespace:

 bearophile proposes the following syntax:

In this case the semantics matters much more than the syntax :-) Bye, bearophile
Jan 18 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 I don't think a new syntax is required. We already have the 
 template syntax:

    RangedInt!(0,10)

 should do it.

Is this array literal accepted, and can D spot the out-of-range bug at compile time (Ada language allows both things)? RangedInt!(0, 10)[] arr = [1, 5, 12, 3, 2]; Probably there are other semantic details that should be handled. Bye, bearophile
Jan 18 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Michel Fortin:

 In fact, even the most obvious case can't be caught at 
 compile-time with the template approach:

 	void baz()
 	{
 		foo(999); // runtime error here
 	}

The constructor of the ranged int needs an "enum precondition": http://forum.dlang.org/thread/ksfwgjqewmsxsribenzq forum.dlang.org Bye, bearophile
Jan 18 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 The point being, there is a whole universe of subset types. You 
 cannot begin to predict all the use cases, let alone come up 
 with a special syntax for each.

On the other hand there are some groups of types that are both very common, lead to a good percentage of bugs and troubles, and need a refined type semantic & management to be be handled well (so it's hard to implement them as library types). So there are some solutions: 1) Pick the most common types, like pointers/class references, integral values, and few others, and hard code their handling very well in the language. 2) Try to implement some barely working versions using the existing language features. 3) Add enough tools to the language to allow the creation of "good enough" library defined features. (This is hard to do. Currently you can't implement "good enough" not-nullable reference types or ranged integers in D). 4) Give up and accept to use a simpler language, with a simpler compiler, that is more easy to create and develop. Bye, bearophile
Jan 18 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
 1) Pick the most common types, like pointers/class references, 
 integral values, and few others, and hard code their handling 
 very well in the language.

This is what Ada usually does.
 2) Try to implement some barely working versions using the 
 existing language features.

This is what D often has done.
 3) Add enough tools to the language to allow the creation of 
 "good enough" library defined features. (This is hard to do. 
 Currently you can't implement "good enough" not-nullable 
 reference types or ranged integers in D).

This is what some language as ATS tries to do.
 4) Give up and accept to use a simpler language, with a simpler 
 compiler, that is more easy to create and develop.

This is what Go often does (in other cases it hard-codes a solution, like the built-in associative arrays). Bye, bearophile
Jan 18 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 18 January 2014 at 02:59:43 UTC, Walter Bright wrote:
 On 1/17/2014 6:42 PM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 But then you have to define "invalid state",

An unexpected value is an invalid state.

It is only an invalid state for a subsystem, if your code is written to handle it, it can contain it and recover (or disable that subsystem). Assuming that you know that it unlikely to be caused by memory corruption. The problem with being rigid on this definition is that most non-trivial programs are constantly in an invalid state and therefore should not be allowed to even start. Basically you should stop making DMD available, it contains bugs, it is constantly in an invalid state vs the published model. State is not only variables. State is code too. (e.g. state machine). What is the essential difference between insisting on stopping a program with bugs and insisting on not starting a program with bugs? There is no difference. Still, most companies ship software with known non-fatal bugs.
Jan 18 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 Currently you can't implement
 "good enough" not-nullable reference types or ranged integers 
 in D).

This is not at all clear.

A good Ranged should allow syntax like this, and it should catch this error at compile time (with an "enum precondition"): Ranged!(int, 0, 10)[] arr = [1, 5, 12, 3, 2]; It also should use the CPU overflow/carry flags to detect efficiently enough integer overflows on a Ranged!(uint, 0, uint.max) type. It should handle the conversions nicely to the super-type and allow the usage of a ranged int as array index. And array bound tests should be disabled if you are using a ranged size_t that is statically known to be in the interval of the array, because this is one of the main purposes of ranged integrals. And D arrays should have optional strongly-typed index types, as in Ada. Because this makes the code safer, easier to reason about, and even faster (thanks to disabling some now unnecessary array bound tests). Similarly not-nullable pointers and class references have some semantic requirements that are not easy to implement in D today. Bye, bearophile
Jan 18 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 18 January 2014 at 21:05:02 UTC, Walter Bright wrote:
 On 1/17/2014 7:05 AM, Michel Fortin wrote:
 Some more thoughts.

The postfix ? has parsing issues with ?: expressions. Aside from that, non-null is only one case of a universe of types that are subsets of other types. I'd prefer a more general solution.

I was thinking about defining reference types (classes, interfaces) as considered non nullable in safe . It is still possible to encapsulate null in NonNull!T .
Jan 18 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 18 January 2014 at 21:05:02 UTC, Walter Bright wrote:
 On 1/17/2014 7:05 AM, Michel Fortin wrote:
 Some more thoughts.

The postfix ? has parsing issues with ?: expressions. Aside from that, non-null is only one case of a universe of types that are subsets of other types. I'd prefer a more general solution.

I was thinking about defining reference types (classes, interfaces) as considered non nullable in safe . It is still possible to encapsulate null in Nullable!T .
Jan 18 2014
prev sibling next sibling parent "Nicolas Sicard" <dransic gmail.com> writes:
On Saturday, 18 January 2014 at 22:12:09 UTC, bearophile wrote:
 Walter Bright:

 I don't think a new syntax is required. We already have the 
 template syntax:

   RangedInt!(0,10)

 should do it.

Is this array literal accepted, and can D spot the out-of-range bug at compile time (Ada language allows both things)? RangedInt!(0, 10)[] arr = [1, 5, 12, 3, 2];

Even though the syntax would be less lean, D can already to this with templates and/or CTFE quite easily.
Jan 19 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 20 January 2014 at 00:44:21 UTC, Michel Fortin wrote:
 On 2014-01-19 20:07:40 +0000, Timon Gehr <timon.gehr gmx.ch> 
 said:

 On 01/19/2014 01:03 PM, Michel Fortin wrote:
 Actually, 'A?' would implicitly convert to 'A' where the 
 compiler can
 prove control flow prevents its value from being null.

I think the type should be upgraded. i.e.:
 So you can
 dereference it in a branch that checked for null:
 
     class A { int i; void foo(); }
     void bar(A a); // non-nullable parameter
 
     void test(A? a, A? a2)
     {
         a.i++; // error, 'a' might be null
         a.foo(); // error, 'a' might be null
         bar(a); // error, 'a' might be null
 
         if (a)
         {

             a.i++; // valid [...]
             a.foo(); // valid [...]
             bar(a); // valid [...]
         }
     }


That's one way to do it. Note that this means you can't assign null to 'a' inside the 'if' branch. But I wouldn't worry too much about that. I think it'd make a good first implementation. What I expect from a not-null feature is that it starts by being over-restrictive and with time, as the control flow analysis evolves, unnecessary restrictions would be lifted. That's similar to how CTFE and purity became what they are today.

I don't see the point of intruducing a new syntax for nullable, when D typesystem is already powerful enough to provide it as lib.
Jan 19 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 19 January 2014 at 08:41:23 UTC, Timon Gehr wrote:
 This is not a plausible assumption. What you tend to know is 
 that the program is unlikely to fail because otherwise it would 
 not have been shipped, being safety critical.

I feel that this "safety critical" argument is ONE BIG RED HERRING. It is completely irrelevant to the topic which is whether it is useful to have recovery mechanisms for null-pointers or not. It is, but not if you are forced to use them. Nobody suggests that you should be forced to recover from a null dereference? Nevertheless: If your application is safety critical and not proven correct or have not undergone exhaustive testing (all combinations of input) then it most likely is a complex system which is likely to contain bugs. You can deal with this by partitioning the system into independent subsystems (think functional programming) which you in a domain specific manner control (e.g. you can have multiple algorithms and select the median value or the most conservative estimate, spinning down subsystems reverting to a less optimal state (more resource demanding), running a verifier on the result etc etc.
 I.e. when it fails, you don't know that it is unlikely to be 
 caused by something. It could be hardware failure, and even a 
 formal correctness proof does not help with that.

But hardware failure is not a valid issue when discussing programming language constructs? Of course the system design should account for hardware failure, extreme weather that makes sensors go out of range and a drunken sailor pressing all buttons at once. Not a programming language construct topic.
 Irrelevant. He is arguing for stopping the system once it has 
 _become clear_ that the _current execution_ might not deliver 
 the expected results.

Then you have to DEFINE what you mean by expected results. Which is domain specific, not a programming language construct issue in a general programming language. If the expected result is defined be: having a result is better than no result, then stopping the system is the worst thing you could do. If the expected result controls N different systems then it might be better to fry 1 system and keep N-1 systems running than to fry N systems. That's a domain specific choice the system designer should have the right to make. Sacrifice one leg to save the other limbs. Think about the effect of this: 1 router detects a bug, by the logic in this thread it should then notify all routers running the same software and tell them to shut down immediately. Result: insta-death to entire Internet.
Jan 20 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 20 January 2014 at 19:27:31 UTC, Walter Bright wrote:
 I infer you think that my arguments here are based on something 
 I dreamed up in 5 minutes of tip-tapping at my keyboard. They 
 are not. They are what Boeing and the aviation industry use 
 extremely successfully to create incredibly safe airliners, and 
 the track record is there for anyone to see.

No, but I think you are conflating narrow domains with a given practice and broader programming application development needs and I wonder what the relevance is to this discussion of having other options than bailing out. I assume that you were making a point that is of relevance to application programmers?? I assume that there is more to this than an anecdote? And… no it is not ok to say that one should accept a buggy program to be in a inconsistent state until the symptoms surface and only then do a headshot, which is the reasoning behind one-buggy-implementation-running-until-null-crash. "you aren't ill until you look ill"? But that is not what Boeing does, is it? Because they use a comparator: "we may be ill, but we are unlikely to show the same symptoms, so if we agree we assume that we are well." Which is a much more acceptable "excuse" (in terms of probability of damage). Why? Because a 0.001% chance of implementation related failure could be reduced to say 0.000000001% (depending on the resolution of the output etc). Ideally safe-D should conceptually give you isolates so that an application can call a third party library that loads a corrupted file and crash on a null-ptr (because that code path has never been run before) and you catch that crash and continue. Yes, the library is buggy and only handles consistent files well, but as an application programmer that is fine. I only want to successfully load non-corrupt files, there is no need to fix that library. Iff the library/language assures that it behaves like it is run as an isolate (no side effects on the rest of the program). Wasting resources on handling corrupt files gracefully is pointless if you can isolate and contain the problem. It is fine if HALT is the default in D, defaults should be conservative. It is not fine if the attitude is that HALT is the best option if the programmer thinks otherwise and anticipates trouble.
Jan 20 2014
prev sibling next sibling parent "Tobias Pankrath" <tobias pankrath.net> writes:
On Monday, 20 January 2014 at 20:01:58 UTC, Ola Fosheim Grøstad 
wrote:
 Ideally safe-D should conceptually give you isolates so that an 
 application can call a third party library that loads a 
 corrupted file and crash on a null-ptr (because that code path 
 has never been run before) and you catch that crash and 
 continue. Yes, the library is buggy and only handles consistent 
 files well, but as an application programmer that is fine.

The point is: for true isolation you'll need another process. If you are aware that it could die: let it be. Just restart it or throw the file away or whatever. So given true isolation hlt on null ptr dereference isn't an issue.
Jan 20 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 20 January 2014 at 20:11:33 UTC, Tobias Pankrath wrote:
 The point is: for true isolation you'll need another process. 
 If you are aware that it could die: let it be. Just restart it 
 or throw the file away or whatever.

That is not an option. I started looking at D in early 2006 because I was looking for a language to create an experimental virtual world server. C++ is out of the question because of all the deficiencies (except for some simulation parts that have to be bug-free) and D could have been a good fit. Forking is too slow and tedious. File loading was just an example. The "isolate" should have read access to global state (measured in gigabytes), but not write access. If you cannot have "probable" isolates in safe D, then it is not suitable for "application level" server designs that undergo evolutionary development. I am not expecting true isolates, but "probable". Meaning: the system is more likely to go down for some other reason than a leaky isolate. Isolates and futures are also very simple and powerful abstractions for getting multi-threading in web services in a trouble free manner.
 So given true isolation hlt on null ptr dereference isn't an 
 issue.

You don't need hardware isolation to do this in a way that works in practice. It should be sufficient to do static analysis and get a list of trouble areas which you can inspect.
Jan 20 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Friday, 17 January 2014 at 19:43:58 UTC, Walter Bright wrote:
 I've almost never had a problem tracking down the cause of a 
 null pointer. Usually just a few minutes with a debugger and 
 getting a backtrace.

It's not always easy to run a debugger on it. I just had an exception (not even a segfault) take two days to track down since, 1) the std.variant exception messages SUCK and 2) after finding where the exception was coming from, it was still hard to find why the input was weird. A bad value from a third party server (generated with the one specific user's request) got stored and used a bit later. The problem wasn't where it was used: it was where it came from. And that's what the NotNull seeks to help.
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 17 January 2014 at 19:43:58 UTC, Walter Bright wrote:
 I've almost never had a problem tracking down the cause of a 
 null pointer. Usually just a few minutes with a debugger and 
 getting a backtrace.

Doesn't work if the unexpected "null" sits in a graph and the source of it is hard to pinpoint or occurs "randomly". E.g. if you are using a "black box" framework or it happens spuriously on a server because it is triggered by a database timeout which never happens on the dev server.
Jan 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 17, 2014 at 11:43:59AM -0800, Walter Bright wrote:
[...]
 I've almost never had a problem tracking down the cause of a null
 pointer. Usually just a few minutes with a debugger and getting a
 backtrace.

I think this depends on the kind of code you write. In callback-heavy code, usually when you're multiplexing between many simultaneous request/response chains, these kinds of problems are very hard to track down. You'll see the null pointer somewhere in your callback's context structure, but no amount of backtrace will help you go any further because they all end at the event dispatch loop pretty shortly up the stack, which doesn't tell you where in the chain of events the null came from. The callback could've been invoked from any number of places (usually callbacks are factored into generic functions so that you don't have to deal with writing 500 callbacks just to handle the response to each event type), so you have to deduce which of the n number of possibilities may have caused it. Usually, that only leads to finding that the previous event handler was only passing things along, so you have to go back yet another step in the chain. When there are n possible ancestors for each step in the chain, you're talking about n^k possible paths to investigate before you find the ultimate culprit. A few minutes is not going to suffice, even for moderate values of k. If you're very very lucky, you may have a log of events that help narrow k to a sufficiently small number that allows quick deduction of where the problem is. Often, though, all you have is a stack trace from an inaccessible customer production server, and you'll have to explore n^k possibilities before you can find the problem. (Even a stack trace should already be counted as lucky; I've had to fix *race conditions* with no stack trace and only indirect evidence that a daemon died and got restarted by init. It took months before we could even reproduce the problem -- it was highly dependent on precise network timing -- much less guess at what went wrong.) Nipping the null at the bud (i.e., know where in the code it was first set as null) is a real life-saver in these kinds of situations. T -- Кто везде - тот нигде.
Jan 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 17, 2014 at 01:29:53PM -0800, Walter Bright wrote:
 On 1/17/2014 1:10 PM, H. S. Teoh wrote:
On Fri, Jan 17, 2014 at 11:43:59AM -0800, Walter Bright wrote:
[...]
I've almost never had a problem tracking down the cause of a null
pointer. Usually just a few minutes with a debugger and getting a
backtrace.

I think this depends on the kind of code you write. In callback-heavy code, usually when you're multiplexing between many simultaneous request/response chains, these kinds of problems are very hard to track down. You'll see the null pointer somewhere in your callback's context structure, but no amount of backtrace will help you go any further because they all end at the event dispatch loop pretty shortly up the stack, which doesn't tell you where in the chain of events the null came from.

What you do then is go back as far as practical, then put asserts in.

Of course, but that requires a few rounds of recompilation, repacking the EEPROM image, installation on device, and repeating a user interaction that possibly only randomly hits the bug. Hardly the work of a few minutes.
When there are n possible ancestors for each step in the chain,
you're talking about n^k possible paths to investigate before you
find the ultimate culprit. A few minutes is not going to suffice,
even for moderate values of k.

Put asserts in, maybe a printf too. I do it all the time to track down problems. Memory corruption can be a difficult one to track down, but I've never had trouble tracking down a null pointer's source. They aren't any harder than any "I have a bad value in this field, I wonder who set it" problem. I.e. it isn't special.

True. But in many cases, the source of a bad value is easy to find because of its uniqueness (hmm, why does variable x have a value of 24576? oh I know, search for 24576 in the code). Nulls, though, can occur in many places, and in D they are implicit. There's no easy way to search for that.
Nipping the null at the bud (i.e., know where in the code it was
first set as null) is a real life-saver in these kinds of situations.

It's better for any bug to catch it at compile time.

Currently D doesn't allow catching nulls at compile-time. T -- Give me some fresh salted fish, please.
Jan 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 17, 2014 at 03:37:22PM -0800, Walter Bright wrote:
 On 1/17/2014 2:05 PM, H. S. Teoh wrote:
Of course, but that requires a few rounds of recompilation, repacking
the EEPROM image, installation on device, and repeating a user
interaction that possibly only randomly hits the bug. Hardly the work
of a few minutes.

I've done EEPROM programming & debugging, I know how to do it. Nulls still aren't a special kind of bug.

I wasn't arguing it is. I'm just saying it's not as trivial as you made it sound.
 BTW, even back in the 1970's, people developing EEPROM systems would
 plug a special device into the board which would redirect the EEPROM
 access to some off-board RAM, or would debug using emulators. Maybe
 this technology has been forgotten :-)

Well, nowadays people use VMware, which is essentially a glorified emulator. :) There are times when you have to test on the actual device, though, when emulation is imperfect.
But in many cases, the source of a bad value is easy to find because
of its uniqueness (hmm, why does variable x have a value of 24576? oh
I know, search for 24576 in the code). Nulls, though, can occur in
many places, and in D they are implicit. There's no easy way to
search for that.

Come on. Every type in D has a default initializer. There's still nothing special about null. Even if you got rid of all the nulls and instead use the null object pattern, you're not going to find it any easier to track it down, and you're in even worse shape because now it can fail and you may not even detect the failure, or may discover the error much, much further from the source of the bug.

If the null object remembers who created it (file + line), this would short-circuit a lot of time-consuming detective work you'd have to do otherwise. That was the main thrust behind my original post. I didn't really find the null object pattern, as it is generally used (dummy values for fields, no-ops for methods), particularly useful as a programming technique. It is, after all, just the OO version of a sentinel, and sentinels have been around long before OO was. But if the null object can be made smarter -- remember which piece of code created it, for example -- then it might begin to be useful. Otherwise, I don't see much value in Andrei's proposal. T -- We've all heard that a million monkeys banging on a million typewriters will eventually reproduce the entire works of Shakespeare. Now, thanks to the Internet, we know this is not true. -- Robert Wilensk
Jan 17 2014
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
--047d7bea2e0cfc891604f03357fb
Content-Type: text/plain; charset=ISO-8859-1

On 17 Jan 2014 01:45, "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org>
wrote:
 Walter and I were talking today about the null pointer issue and he had

 One common idiom to replace null pointer exceptions with milder

that is used in lieu of the null reference to initialize all otherwise uninitialized references. In D that would translate naturally to:
 class Widget
 {
     private int x;
     private Widget parent;
     this(int y) { x = y; }
     ...
     // Here's the interesting part
     static Widget init = new Widget(42);
 }

 Currently the last line doesn't compile, but we can make it work if the

allocate a static buffer for the "new"ed Widget object and will make init point there.
 Whenever a Widget is to be default-initialized, it will point to

because currently (with no init definition) Widget.init is null.
 So the init Widget will satisfy:

 assert(x == 42 && parent is Widget.init);

 Further avenues are opened by thinking what happens if e.g. init is

 Thoughts?

I would have thought that sort of code would be doable since someone thought it be a good idea to introduce ClassReferenceExp to the frontend implementation. :) --047d7bea2e0cfc891604f03357fb Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">On 17 Jan 2014 01:45, &quot;Andrei Alexandrescu&quot; &lt;<a= href=3D"mailto:SeeWebsiteForEmail erdani.org">SeeWebsiteForEmail erdani.or= g</a>&gt; wrote:<br> &gt;<br> &gt; Walter and I were talking today about the null pointer issue and he ha= d the following idea.<br> &gt;<br> &gt; One common idiom to replace null pointer exceptions with milder reprod= ucible errors is the null object pattern, i.e. there is one object that is = used in lieu of the null reference to initialize all otherwise uninitialize= d references. In D that would translate naturally to:<br> &gt;<br> &gt; class Widget<br> &gt; {<br> &gt; =A0 =A0 private int x;<br> &gt; =A0 =A0 private Widget parent;<br> &gt; =A0 =A0 this(int y) { x =3D y; }<br> &gt; =A0 =A0 ...<br> &gt; =A0 =A0 // Here&#39;s the interesting part<br> &gt; =A0 =A0 static Widget init =3D new Widget(42);<br> &gt; }<br> &gt;<br> &gt; Currently the last line doesn&#39;t compile, but we can make it work i= f the respective constructor is callable during compilation. The compiler w= ill allocate a static buffer for the &quot;new&quot;ed Widget object and wi= ll make init point there.<br> &gt;<br> &gt; Whenever a Widget is to be default-initialized, it will point to Widge= t.init (i.e. it won&#39;t be null). This beautifully extends the language b= ecause currently (with no init definition) Widget.init is null.<br> &gt;<br> &gt; So the init Widget will satisfy:<br> &gt;<br> &gt; assert(x =3D=3D 42 &amp;&amp; parent is Widget.init);<br> &gt;<br> &gt; Further avenues are opened by thinking what happens if e.g. init is pr= ivate or disable-d.<br> &gt;<br> &gt; Thoughts?</p> <p dir=3D"ltr">I would have thought that sort of code would be doable since= someone thought it be a good idea to introduce ClassReferenceExp to the fr= ontend implementation. :)</p> --047d7bea2e0cfc891604f03357fb--
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 17 January 2014 at 21:31:16 UTC, Walter Bright wrote:
 As I replied elsewhere, tracking down the source of a bad value 
 in any variable is a standard debugging problem. There isn't 
 anything special about null in this regard.

It isn't special in the theoretical sense, but since it in practice is used for a wide variety of things it becomes a sensitive issue in situations where you cannot replay input. Especially since it is used for linking together subsystems. In multi user games you can do fine with wrong "float values", objects might drift, you might get odd effects, but those bugs can be cool. They might even become features. A wrong "null" makes the client or server shut down, it is disruptive to the service. So what can you do? You can log all input events in a ring buffer and send a big dump to the developers when the game client crash and try to find correlation between stack trace and input events (if you get multiple reports). And on a server with say 1000 concurrent users that interact and trigger a variety of bugs in one big multi-threaded global state world sim… You want to use a safe language for world content, where you disable features if needed, but keep the world running. The same goes for a web service. You don't want to shut down the whole service just because the "about page" handler fails to test for null. Big systems have to live with bugs, it is inevitable that they run with bugs. Erlang was created for stuff like that. Can you imagine a telephone central going down just because the settings for one subscriber was wrong and triggered a bug?
Jan 17 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 18 January 2014 at 01:05:50 UTC, Walter Bright wrote:
 Because I've tracked down the cause of many, many null 
 pointers, and I've tracked down the cause of many, many other 
 kinds of invalid values in a variable. Null pointers tend to 
 get detected much sooner, hence closer to where they were set.

To be fair, I do think we should require explicit initialization for char for instance instead of putting in them an invalid codepoint. Still, the consequences for null are much more dramatic, which make it more worthwhile to change. They don't have the same ROI as they don't have the same cost.
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 18 January 2014 at 01:05:50 UTC, Walter Bright wrote:
 Because I've tracked down the cause of many, many null 
 pointers, and I've tracked down the cause of many, many other 
 kinds of invalid values in a variable. Null pointers tend to 
 get detected much sooner, hence closer to where they were set.

Some null pointers. The ones that were set in the current call-chain, but not the long-lived ones. Anyway, pointers are special since they link together subsystems. Yes, it is better to have a null-value than a pointer to a random location, so "null" isn't the worst value for a pointer. It is the best arbitrary wrong value, but there is nothing wrong with saying "Hey, when we are wrong about User identity, we want to keep the system running assuming a Guest identity". However C-like null values conflate a wide array of semantics, I think you can have near half a dozen types of null values in a database. In c-like languages null can mean: not-yet-initialized (used too early), failed-computation (bottom), entity-shall-not-have-subentity (deliberate lack of), empty-container (list is empty), service-not-ready (try again), etc. So yeah, it would be nice if the language is able to distinguish between the different semantics of "null" of different pointer types and convert them into something sensible when "cast" to a different type. For some pointer types it might make sense to kill the thread, for others it would make sense to throw an exception, for others it might make sense to allocate an empty object, for others it might make sense to use a dummy/sentinel. It isn't universally true that the best option is to core dump, but you could do both. You could fork(), core-dump the fork, and recover the original.
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 18 January 2014 at 01:46:55 UTC, Walter Bright wrote:
 The autopilot software was designed by someone who thought it 
 should keep operating even if it detects faults in the software.

I would not write autopilot or life-support software in D. So that is kind of out-of-scope for the language. But: Keep the system simple, select a high level language and verify correctness by an automated proof system. Use 3 independently implemented systems and shut down the one that produces deviant values. That covers more ground than the unlikely null-pointers in critical systems. No need to self-detect anything.
 Consider also the Toyota. My understanding from reading reports 
 (admittedly journalists botch up the facts) is that a single 
 computer controls the brakes, engine, throttle, ignition 
 switch, etc. Oh joy. I wouldn't want to be in that car when it 
 keeps on going despite having self-detected faults.

So you would rather have the car drive off the road because the anti-skid software abruptly turned itself off during an emergency manoeuvre? But would you stay in a car where the driver talks in a cell-phone while driving, or would you tell him to stop? Probably much more dangerous if you measured correlation between accidents and system features. So you demand perfection from a computer, but not from a human being that is exhibiting risk-like behaviour. That's an emotional assessment. The rational action would be to improve the overall safety of the system, rather than optimizing a single part. So spend the money on installing a cell-phone jammer and an accelerator limiter rather than investing in more computers. Clearly, the computer is not the weakest link, the driver is. He might not agree, but he is and he should be forced to exhibit low risk behaviour. Direct effort to where it has most effect. (From a system analytical point of view. It might not be a good sales tactic, because car buyers aren't that rational.)
Jan 17 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Ola Fosheim Grøstad:

 I would not write autopilot or life-support software in D. So 
 that is kind of out-of-scope for the language.

Do you want to use Ada for those purposes? Bye, bearophile
Jan 17 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 18 January 2014 at 01:46:55 UTC, Walter Bright wrote:
 On 1/17/2014 4:44 PM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 Big systems have to live with bugs, it is inevitable that they 
 run with bugs.

It's a dark and stormy night. You're in a 747 on final approach, flying on autopilot. Scenario 1 ---------- The autopilot software was designed by someone who thought it should keep operating even if it detects faults in the software. The software runs into a null object when there shouldn't be one, and starts feeding bad values to the controls. The plane flips over and crashes, everybody dies. But hey, the software kept on truckin'! Scenario 2 ---------- The autopilot software was designed by Boeing. Actually, there are two autopilots, each independently developed, with different CPUs, different hardware, different algorithms, different languages, etc. One has a null pointer fault. A deadman circuit sees this, and automatically shuts that autopilot down. The other autopilot immediately takes over. The pilot is informed that one of the autopilots failed, and the pilot immediately shuts off the remaining autopilot and lands manually. The passengers all get to go home. Note that in both scenarios there are bugs in the software. Yes there have been incidents with earlier autopilots where bugs in it caused the airplane to go inverted. Consider also the Toyota. My understanding from reading reports (admittedly journalists botch up the facts) is that a single computer controls the brakes, engine, throttle, ignition switch, etc. Oh joy. I wouldn't want to be in that car when it keeps on going despite having self-detected faults. It could, you know, start going at full throttle and ignore all signals to brake or turn off, only stopping when it crashes or runs out of gas.

You are running a huge website. Let's say for instance a social network with more than a billion users. Scenario 1 ---------- The software was designed by someone who thought it should keep operating even if it detects faults in the software. A bug arise in some fronted and it start corruption data. Some monitoring detects the issue, the code get fixed and corrupted data are recovered from backup. Users that ended up on that cluster saw their account not working for a day, but everything is back to normal the day after. Scenario 2 ---------- The software was designed by an ex employee from boeing. He know that he should make his software crash hard and soon. As soon as some error are detected on a cluster, the cluster goes down. Hopefully, no data is corrupted, but the load on that cluster must now be handled by other cluster. Soon enough, these clusters overload and the whole website goes down. Hopefully, no data were corrupted in the process, so it isn't needed to restore anything from backup. Different software, different needs. Ultimately, that distinction is irrelevant anyway. The whole possibility of these scenarios can be avoided in case of null dereferences by proper language design.
Jan 17 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 18, 2014 at 02:22:22AM +0000, digitalmars-d-bounces puremagic.com
wrote:
 On Saturday, 18 January 2014 at 01:46:55 UTC, Walter Bright wrote:

Consider also the Toyota. My understanding from reading reports
(admittedly journalists botch up the facts) is that a single
computer controls the brakes, engine, throttle, ignition switch,
etc. Oh joy. I wouldn't want to be in that car when it keeps on
going despite having self-detected faults.

So you would rather have the car drive off the road because the anti-skid software abruptly turned itself off during an emergency manoeuvre?

You missed his point. The complaint is that the car has a *single* software system that handles everything. That's a single point of failure. When that single software system fails, *everything* fails. A fault-tolerant design demands at least two anti-skid software units, where the redundant unit will kick in when the primary one turns off or stops for whatever reason. So when a software fault occurs in the primary unit, it gets shut off, and the backup unit takes over and keeps the car stable. You'd only crash in the event that *both* units fail at the same time, which is far less likely than a single unit failing. This is better than having a single software system that tries to fix itself when it goes wrong, because the fact that something caused part of the code to crash (segfault, or whatever) is a sign that the system is no longer in a state anticipated by the engineers, so there's no guarantee it won't make things worse when it tries to fix itself. For example, it might be scrambled into a state where it keeps the accelerator on with no way to override it, thereby making the problem worse. You need a *decoupled* redundant system to be truly confident that whatever fault caused the problem in the first system doesn't also affect the backup / self-repair system, something which doesn't hold for a single software unit (for example, if the power supply to the unit fails, then whatever self-repair subsystem it has will also be non-functional). That way, when the first unit goes wrong, it can simply be shut off safely, thereby preventing making the problem worse, and the backup unit takes over and keeps things going. To use a software example: if you have a single process that tries to fix itself when, say, a null pointer is dereferenced, then there's no guarantee that the error recovery code won't do something stupid, like format your disk (because the null pointer in an unexpected place proves that the code has logic problems: it isn't in a state that the engineers planned for, so who knows what else is wrong with it -- maybe a function pointer to display graphics has been accidentally replaced with a pointer to the formatDisk function due to the bug that caused the null to appear in an unexpected place). If instead you have two redundant processes, one of which is doing the real work and the second is just sleeping, then when the first process segfaults due to a null pointer, the second one can kick into action -- since it hasn't been doing the same thing as the first process, it's likely still in a safe, consistent state, and so it can safely take over and keep the service running. This is the premise of high-availability systems: there's a primary server that's doing the real work, and one or more redundant units. When the primary dies (power loss, CPU overheat, segfault causing it no longer to respond, etc.), a watchdog timer triggers a failover to the second unit, thus minimizing service interruption time. The failover detection code can then contact an administrator (email, SMS, etc.) notifying that something went wrong with the first unit, and service continues uninterrupted while the first unit is repaired. OTOH, if you have only a single unit and something goes wrong, there's a risk that the recovery code will go wrong too, so the entire unit stops functioning, and service is interrupted until it's repaired. T -- Study gravitation, it's a field with a lot of potential.
Jan 17 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 18 January 2014 at 02:48:38 UTC, Walter Bright wrote:
 I didn't mention that the dual autopilots also have a 
 comparator on the output, and if they disagree they are both 
 shut down. The deadman is an additional check. The dual system 
 has proven itself, a third is not needed.

The pilot is engaged as the third. There are situations where you cannot have a third "intelligent" agent take over, so you should have 3 systems, and reboot and resync the one that diverges, but this is rather off topic. I don't think D is a language that should be used for these kind of systems.
 Please reread what I wrote. I said it shuts itself off and 
 engages the backup, and if there is no backup, you have failed 
 at designing a safe system.

A car driver that is doing an emergency manoeuvre is not part of a safe system, indeed! If you want one system to take over for another you need a safe spot to do it in. Just disappearing instantly isn't optimal because instantly changing responsiveness is a gurantee for failure. In fact, being instantly disruptive is usually the wrong thing to do. You should spin down gracefully. I don't see why you cannot do that with null-pointers. You obviously can do it with division by zero errors. I think you associate null-pointers with memory corruption, which truly is an invalid state for which you might want to instantly shut down.
 I have experience with this stuff, Ola, from my years at Boeing 
 designing flight critical systems. What I outlined is neither 
 irrational nor emotionally driven, and has the safety record to 
 prove its effectiveness.

In a very narrow field where the pilot is monitoring the system and can take over. The pilot is the ultimate source for failure (in a political sense). So you basically shut down the technology and blame the pilot if you end up with a crash. That only works if the computer has been made to replace a human being.
Jan 17 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Saturday, 18 January 2014 at 06:10:20 UTC, Walter Bright wrote:
 On 1/17/2014 7:38 PM, Walter Bright wrote:
 But I agree that compile time
 detect of null reference bugs is better than runtime detection 
 of them.

BTW, the following program: class C { int a,b; } int test() { C c; return c.b; } When compiled with -O: foo.d(6): Error: null dereference in function _D3foo4testFZi It isn't much, only working on intra-function analysis and only when the optimizer is used, but it's something. It's been in dmd for a long time.

But: ---- class C { int a, b; } C create() pure nothrow { return null; } int test() pure nothrow { C c = create(); return c.b; } void main() { test(); } ---- Print nothing, even with -O. Maybe the idea to use C? for nullable references would be worth to implement.
Jan 18 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 18 January 2014 at 03:07:30 UTC, H. S. Teoh wrote:
 You missed his point. The complaint is that the car has a 
 *single*
 software system that handles everything. That's a single point 
 of
 failure. When that single software system fails, *everything* 
 fails.

I didn't miss the point at all. My point is that you should always target the cost of improving the statistical overall safety of the system rather than optimizing the stability of a single part that almost never fail. Having multiple independent software implementations only works for very simple systems. And in that case you can prove correctness by formal proofs. It is more likely to fail due to a loose wire or electrical components.
Jan 18 2014
prev sibling next sibling parent "Tobias Pankrath" <tobias pankrath.net> writes:
On Saturday, 18 January 2014 at 11:03:22 UTC, Jacob Carlborg 
wrote:
 On 2014-01-18 11:41, Daniel Murphy wrote:

 The code that detects the null dereference is in the optimizer.

Why is it located there?

It's probably flow sensitive and only the optimizer builds a CFG.
Jan 18 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi gmail.com> writes:
On Saturday, 18 January 2014 at 00:44:56 UTC, Ola Fosheim Grøstad 
wrote:
 Big systems have to live with bugs, it is inevitable that they 
 run with bugs. Erlang was created for stuff like that. Can you 
 imagine a telephone central going down just because the 
 settings for one subscriber was wrong and triggered a bug?

Stupid me, that I thought that all this mess was solved decades ago with the CPU protected mode. I tend to agree with Walter on this front. -- Paolo
Jan 18 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
I somehow missed this post, so a delayed response.

On Saturday, 18 January 2014 at 01:22:48 UTC, Walter Bright wrote:
 First off, in all these scenarios you describe, how does not 
 having a null make it EASIER to track down the bug?

I have not argued for not having a null. I have argued for trapping null, instantiating a type specific default and recover if the type's meta says so. That default could be to issue a NullException. At that point you should be able to log null dereferences too. I have previously argued that non-null pointers is nice, but only if you have whole program analysis. Otherwise you'll end up injecting null-tests everywhere.
 there's a bug in the code. It's quite another to have a 
 critical system start behaving erratically because of a bug.

Yes, and this is a problematic definition since "erratic" is kind of subjective given that most systems don't follow the model 100%.
 do. The correct way is when the bug is detected, that software 
 is IMMEDIATELY SHUT DOWN and the backup is engaged. If you 
 don't have such a backup, you have a very, very badly designed 
 system.

In that case 99.99% of all software is very, very badly designed. DMD inclusive. Human beings too, but we happen to be fault tolerant, not by using backups, but by being good at recovery and dealing in a fuzzy way with incomplete or wrong information.
Jan 18 2014
prev sibling next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Sunday, 19 January 2014 at 02:33:15 UTC, Walter Bright wrote:
 You elided the qualification "If it is a critical system". dmd 
 is not a safety critical application.

Nor are 99.99%, possibly 100%, of the applications currently being built with D. The points are valid, no safety critical application should ever rely on any individual components not failing. But for the purpose of D, they are not particularly applicable. People are just arguing over two different things (safety critical code vs standard D code), with neither disagreeing with the others, simply bringing up different situations.
Jan 18 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jacob Carlborg:

 I think nil (null) works quite nicely in Ruby. nil is a 
 singleton instance of the NilClass class. Since it's an object 
 you can call methods on it, like to_s, which returns an empty 
 string. It works quite well when doing web development with 
 Ruby on Rails. If you're trying to render something that's nil 
 you'll get nothing, instead of crashing that whole page. Sure, 
 there might be a small icon or similar that isn't rendered but 
 that's usually a minor detail. If the page is working and main 
 content is rendered that's preferable.

Walter is arguing against this solution in D. While that can be OK for Ruby used for web development, for a statically typed language meant for safe coding styles, there are better type-based solution to that problem. Bye, bearophile
Jan 19 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 19 January 2014 at 07:40:09 UTC, Walter Bright wrote:
 On 1/18/2014 6:33 PM, Walter Bright wrote:
 You elided the qualification "If it is a critical system". dmd 
 is not a safety critical application.

And I still practice what I preach with DMD. DMD never attempts to continue running after it detects that it has entered an invalid state - it ceases immediately. Furthermore, when it detects any error in the source code being compiled, it does not generate an object file.

I think the whole "critical system" definition is rather vague. For safety critical applications you want proven implementation technology, proper tooling and a methodology to go with it. And it is very domain specific. Simple algorithms can be proven correct, some types of signal processing can be proven correct/stable, some types of implementations (like a FPGA) affords exhaustive testing (test all combination of input). In the case of D, I find that a somewhat theoretical argument. D is not a proven technology. D does not have tooling with a methodology to go with it. But yes, you want backups due to hardware failure even for programs that are proven correct. In a telephone central you might want to have a backup system to handle emergency calls. If you take a theoretical position (which I think you do) then I also think you should accept a theoretical argument. And the argument is that there is no theoretical difference between allowing programs with known bugs to run and allowing programs with anticipated bugs to run (e.g. catching "bottom" in a subsystem). There is also no theoretical difference between allowing DMD to generate code that is not following the spec 100%, and allowing DMD to generate code if an anticipated "bottom" occurs. It all depends on what degree of deviance from the specified model you accept. It is quite acceptable to catch "bottom" for an optimizer and generate less optimized code for that function, or to turn off that optimizer setting. However, in a compiler you can defer to "the pilot" (compiler) so that is generally easier. In a server you can't.
Jan 19 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 19 January 2014 at 10:32:36 UTC, Jacob Carlborg wrote:
 I think nil (null) works quite nicely in Ruby. nil is a 
 singleton instance of the NilClass class. Since it's an object 
 you can call methods on it, like to_s, which returns an empty 
 string. It works quite well when doing web development with

I have no experience with Ruby, but Javascript also do this (undefined is an object). I don't think it is more work to debug Python and Javascript null exceptions than C-like code. I think it was a mistake to let zero represent null, and it was probably done to make testing faster. But a different bit-pattern could prevent conflation between memory corruption and null. It is highly improbable that an address like $F1234324 is the result of memory corruption. Another advantage with null-objects as a pattern and mechanisms to support it is that you can have multiple variants and differentiate between: - undefined (not initialized) - null (deliberately not having an attribute) - application specific null values (like, try-again, lazy-evaluation) etc that depending on context evaluates to undefined, null or tries to fetch the values by computation. - ++
Jan 19 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi gmail.com> writes:
On Sunday, 19 January 2014 at 12:29:14 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 19 January 2014 at 10:32:36 UTC, Jacob Carlborg 
 wrote:

 I have no experience with Ruby, but Javascript also do this 
 (undefined is an object). I don't think it is more work to 
 debug Python and Javascript null exceptions than C-like code.

Having had heavy experience with Python, and having used D since D1, I would tell the contrary, that's easier to handle null exception with D. That's only experience based, naturally, I don't pretend this to be science. Going down the hill, we should then love PHP, as it's mantra seems to be "go marching in", and we all know what an horrible mess it is. --- Paolo
Jan 19 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi gmail.com> writes:
On Sunday, 19 January 2014 at 12:20:42 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 19 January 2014 at 07:40:09 UTC, Walter Bright wrote:
 On 1/18/2014 6:33 PM, Walter Bright wrote:
 You elided the qualification "If it is a critical system". 
 dmd is not a safety critical application.

And I still practice what I preach with DMD. DMD never attempts to continue running after it detects that it has entered an invalid state - it ceases immediately. Furthermore, when it detects any error in the source code being compiled, it does not generate an object file.

I think the whole "critical system" definition is rather vague. For safety critical applications you want proven implementation technology, proper tooling and a methodology to go with it. And it is very domain specific. Simple algorithms can be proven correct, some types of signal processing can be proven correct/stable, some types of implementations (like a FPGA) affords exhaustive testing (test all combination of input). In the case of D, I find that a somewhat theoretical argument. D is not a proven technology. D does not have tooling with a methodology to go with it. But yes, you want backups due to hardware failure even for programs that are proven correct. In a telephone central you might want to have a backup system to handle emergency calls. If you take a theoretical position (which I think you do) then I also think you should accept a theoretical argument. And the argument is that there is no theoretical difference between allowing programs with known bugs to run and allowing programs with anticipated bugs to run (e.g. catching "bottom" in a subsystem). There is also no theoretical difference between allowing DMD to generate code that is not following the spec 100%, and allowing DMD to generate code if an anticipated "bottom" occurs. It all depends on what degree of deviance from the specified model you accept. It is quite acceptable to catch "bottom" for an optimizer and generate less optimized code for that function, or to turn off that optimizer setting. However, in a compiler you can defer to "the pilot" (compiler) so that is generally easier. In a server you can't.

I'm trying to understand your motivations, but why in a server you can't? I still can't grasp that point. -- Paolo
Jan 19 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 19 January 2014 at 14:23:44 UTC, Paolo Invernizzi 
wrote:
 However, in a compiler you can defer to "the pilot" (compiler) 
 so that is generally easier. In a server you can't.

I'm trying to understand your motivations, but why in a server you can't? I still can't grasp that point.

It was a typo: However, in a compiler you can defer to "the pilot" (programmer) so that is generally easier. In a server you can't.
Jan 19 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 19 January 2014 at 14:14:11 UTC, Paolo Invernizzi 
wrote:
 Having had heavy experience with Python, and having used D 
 since D1, I would tell the contrary, that's easier to handle 
 null exception with D.

Leaving dynamic vs non-dynamic aspects aside, why do you find null dereferencing easier to handle (i.e. debug) in D than in Python?
Jan 19 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 19 January 2014 at 19:55:34 UTC, Walter Bright wrote:
 On 1/19/2014 4:20 AM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 If you take a theoretical position (which I think you do)

Please set such statements against my assertion that Boeing follows the principles I outlined in practice, and very successfully.

Would they even consider using D? Are you referring to a niche for which D is not suited? In most situations you cannot afford to develop a program to the point where it is 100% bug free, let alone develop 2 independent versions of the spec to 100% bug free. This is a very narrow niche. And it isn't even relevant, because crash-and-burn-on-null is still an option with voluntary recovery mechanisms. And as I pointed out, you wouldn't use zero as a null value if safety and debugging is your prime concern. You would use a more "secure" bitpattern (i.e. harder to arrive at by accident), and you would differentiate "null" from "undefined"?
Jan 19 2014