• Davidson Corry <davidsoncorry comcast.net> Apr 30 2008
• "Bruce Adams" <tortoise_74 yeah.who.co.uk> Apr 30 2008
• Davidson Corry <davidsoncorry comcast.net> May 01 2008
• Davidson Corry <davidsoncorry comcast.net> May 01 2008
• Jesse Phillips <jessekphillips gmail.com> May 01 2008
• Davidson Corry <davidsoncorry comcast.net> May 02 2008
• "Me Here" <p9e883002 sneakemail.com> May 02 2008
• Walter Bright <newshound1 digitalmars.com> May 02 2008
• Davidson Corry <davidsoncorry comcast.net> May 02 2008
• Russell Lewis <webmaster villagersonline.com> May 02 2008
• Russell Lewis <webmaster villagersonline.com> May 03 2008
• "Janice Caron" <caron800 googlemail.com> May 02 2008

I have been following, fascinated, the discussions of keywords such as 
pure, const, invariant and Janice Caron's elegantly evocative 
"paintable". It occurred to me the other day to wonder if all this is 
heading in the wrong direction.

Memory management is difficult and fragile. Many keywords, language 
constructs and library routines were introduced to support it, such as 
malloc/free, new/delete, dispose, Set obj = Nothing etc. But it remained 
difficult and fragile... until garbage collection was built into 
languages, and we just forgot about it. Now nine times out of ten we 
don't use memory management keywords at all [1], and when we do, nine 
times out of ten we shouldn't have.

Are keywords for availability -- const, invariant, synchronized, etc. -- 
awaiting the same sort of epiphany?

 --------------

An object must be responsible for setting and for reporting its own 
state. No outside client can be allowed to change the object's member 
data, nor to set the algorithms by which those data are calculated and 
reported. However, the object can provide accessor methods through which 
a client can request the /object/ do to so itself -- set, get, etc. [2]

At various times, an object might be in one of three conditions of 
availability: [3]
 - changeable (clients can call methods which will modify the object's 
state)
 - inspectable (clients can call methods which report the object's state 
but do not change it)
 - detectable (clients can determine that the object exists, but not 
inspect its state nor change it)
For instance, while executing a state-modifier method, the object's 
consistency guarantees - its class invariant - is temporarily suspended. 
So the object cannot allow potential clients on other threads even to 
inspect it, much less to change it. But it should allow other threads to 
detect its existence so that they can decide whether or not to wait for 
it to get its act together.

An object's availability may change from time to time during the 
execution of the program. An object may choose to be changeable at some 
point, and then later "lock itself down" so that it is inspectable but 
denies all clients access to its state-modifier methods. And then, 
perhaps, open itself up again. Availability might be considered to be a 
distinct property of all objects, inherited from the uber-parent Object 
class. An object may explicitly set its availability property, /or the 
language might set it automatically at runtime/, or both.

Declaring an object const is the limiting case that the object is only 
inspectable -- that is, makes no modifier-methods available -- 
throughout the lifetime of of the object (or the program). No client can 
ask the object to change itself, although the object may itself choose 
to change. Declaring an object invariant is the same, except that the 
object also promises not to change itself. A compiler may be able to 
take advantage of optimization opportunities for objects that are 
lifetime const or invariant. Similarly, there may be opportunities for 
runtime optimizations on objects that change their availability dynamically.

A language-system could inspect the source code, "drilling down" into 
sub-objects, and infer which objects and methods are effectively const 
or invariant, and (if compiled for multi-threaded or multi-programmed 
access) which need locking and rendezvous support. Explicit keywords are 
not required. However, to do so requires that the language-system have 
access to the source code for the entire system, and is thus in conflict 
with separate compilation, and with coding for libraries. [4] [5]

 --------------------

And now for the Democratic response:

How a program actually behaves at runtime implies a contract for its 
objects. Yet languages which implement design-by-contract allow 
programmers to explicitly /declare/ a contract separate from the implied 
one. And this is an advantage because the implied contract cannot be 
readily perceived by reading the code, and because conflicts between the 
implied and declared contracts can be caught at compile time (e.g. 
Spec#) and at runtime.This simplifies debugging and enhances program 
correctness. So keywords for contracting are justified.

Could keywords (or object properties or library routines automatically 
provided by the language) for constness, purity, synchronization also be 
a double-check on the implied mutability and thread-singularity of 
runnable code? Hmm...

 --------------------

 -- Davidson Corry


[1] C++ new did both memory management and object creation. In Java/C#/D 
it is strictly a creation keyword.
[2] if the language-system chooses to /implement/ an accessor as a 
direct access, fine. Conceptually it's still a method managed by the object.
[3] roughly, mutable, const and locked. But I am trying to avoid using 
actual keywords with specific technical meanings
[4] Or is it? The compiler could embed in the object file, or in the 
library export definition, metadata that declares the relevant 
properties of the public objects, and allows the linker to correctly 
optimize and support whole-system linkage. This pushes what is 
traditionally considered part of the compiler's job into the linker, but 
so what?
[5] It's also in conflict with Walter's stated goal of making D small 
and clean enough for others to implement compilers for it. Oh well, 
there's always the "supported syntactically but not semantically" 
option. ;-)

On Thu, 01 May 2008 05:51:57 +0100, Davidson Corry  =

<davidsoncorry comcast.net> wrote:

 I have been following, fascinated, the discussions of keywords such as=



 pure, const, invariant and Janice Caron's elegantly evocative  =


 "paintable". It occurred to me the other day to wonder if all this is =



 heading in the wrong direction.

 Memory management is difficult and fragile. Many keywords, language  =


 constructs and library routines were introduced to support it, such as=



 malloc/free, new/delete, dispose, Set obj =3D Nothing etc. But it rema=



 difficult and fragile... until garbage collection was built into  =


 languages, and we just forgot about it. Now nine times out of ten we  =


 don't use memory management keywords at all [1], and when we do, nine =



 times out of ten we shouldn't have.

 Are keywords for availability -- const, invariant, synchronized, etc. =



 awaiting the same sort of epiphany?

  --------------

 An object must be responsible for setting and for reporting its own  =


 state. No outside client can be allowed to change the object's member =



 data, nor to set the algorithms by which those data are calculated and=



 reported. However, the object can provide accessor methods through whi=



 a client can request the /object/ do to so itself -- set, get, etc. [2=


 At various times, an object might be in one of three conditions of  =


 availability: [3]
  - changeable (clients can call methods which will modify the object's=



 state)
  - inspectable (clients can call methods which report the object's sta=



 but do not change it)
  - detectable (clients can determine that the object exists, but not  =


 inspect its state nor change it)
 For instance, while executing a state-modifier method, the object's  =


 consistency guarantees - its class invariant - is temporarily suspende=



 So the object cannot allow potential clients on other threads even to =



 inspect it, much less to change it. But it should allow other threads =



 detect its existence so that they can decide whether or not to wait fo=



 it to get its act together.

 An object's availability may change from time to time during the  =


 execution of the program. An object may choose to be changeable at som=



 point, and then later "lock itself down" so that it is inspectable but=



 denies all clients access to its state-modifier methods. And then,  =


 perhaps, open itself up again. Availability might be considered to be =



 distinct property of all objects, inherited from the uber-parent Objec=



 class. An object may explicitly set its availability property, /or the=



 language might set it automatically at runtime/, or both.

 Declaring an object const is the limiting case that the object is only=



 inspectable -- that is, makes no modifier-methods available --  =


 throughout the lifetime of of the object (or the program). No client c=



 ask the object to change itself, although the object may itself choose=



 to change. Declaring an object invariant is the same, except that the =



 object also promises not to change itself. A compiler may be able to  =


 take advantage of optimization opportunities for objects that are  =


 lifetime const or invariant. Similarly, there may be opportunities for=



 runtime optimizations on objects that change their availability  =


 dynamically.

 A language-system could inspect the source code, "drilling down" into =



 sub-objects, and infer which objects and methods are effectively const=



 or invariant, and (if compiled for multi-threaded or multi-programmed =



 access) which need locking and rendezvous support. Explicit keywords a=



 not required. However, to do so requires that the language-system have=



 access to the source code for the entire system, and is thus in confli=



 with separate compilation, and with coding for libraries. [4] [5]

  --------------------

 And now for the Democratic response:

 How a program actually behaves at runtime implies a contract for its  =


 objects. Yet languages which implement design-by-contract allow  =


 programmers to explicitly /declare/ a contract separate from the impli=



 one. And this is an advantage because the implied contract cannot be  =


 readily perceived by reading the code, and because conflicts between t=



 implied and declared contracts can be caught at compile time (e.g.  =


 Spec#) and at runtime.This simplifies debugging and enhances program  =


 correctness. So keywords for contracting are justified.

 Could keywords (or object properties or library routines automatically=



 provided by the language) for constness, purity, synchronization also =



 a double-check on the implied mutability and thread-singularity of  =


 runnable code? Hmm...

  --------------------

  -- Davidson Corry


 [1] C++ new did both memory management and object creation. In Java/C#=



 it is strictly a creation keyword.
 [2] if the language-system chooses to /implement/ an accessor as a  =


 direct access, fine. Conceptually it's still a method managed by the  =


 object.
 [3] roughly, mutable, const and locked. But I am trying to avoid using=



 actual keywords with specific technical meanings
 [4] Or is it? The compiler could embed in the object file, or in the  =


 library export definition, metadata that declares the relevant  =


 properties of the public objects, and allows the linker to correctly  =


 optimize and support whole-system linkage. This pushes what is  =


 traditionally considered part of the compiler's job into the linker, b=



 so what?
 [5] It's also in conflict with Walter's stated goal of making D small =



 and clean enough for others to implement compilers for it. Oh well,  =


 there's always the "supported syntactically but not semantically"  =


 option. ;-)


I agree with pretty much everything you've said. However, I'm not clear =
 =

what point you're trying to make.

Bruce Adams wrote:
 I agree with pretty much everything you've said. However, I'm not clear 
 what point you're trying to make.


I often feel that way myself. <grin>

My point was that the language system ought to manage constness, 
invariance, synchronization on objects shared by multiple threads or 
processes, and so on, AUTOMATICALLY without requiring the programmer to 
manage those things explicitly (via keywords or whatever else), just as 
GC-enabled language systems now automatically manage memory allocation 
without requiring the programmer to use keywords etc. to do memory 
allocation explicitly.

And my counterpoint was to suggest that, if the language system DOES 
provide keywords/properties/libraries by which const etc. CAN be managed 
explicitly if desired (but not required), then those tools should be an 
orthogonal doublecheck on the language's automatic inference of how 
const etc. is to be managed, just as design by contract is a doublecheck 
on the "contract" implied by a program's actual behavior.

-- Dai

Bruce Adams wrote:
 I agree with pretty much everything you've said. However, I'm not clear 
 what point you're trying to make.


I often feel that way myself. <grin>

My point was that the language system ought to manage constness, 
invariance, synchronization on objects shared by multiple threads or 
processes, and so on, AUTOMATICALLY without requiring the programmer to 
manage those things explicitly (via keywords or whatever else), just as 
GC-enabled language systems now automatically manage memory allocation 
without requiring the programmer to use keywords etc. to do memory 
allocation explicitly.

And my counterpoint was to suggest that, if the language system DOES 
provide keywords/properties/libraries by which const etc. CAN be managed 
explicitly if desired (but not required), then those tools should be an 
orthogonal doublecheck on the language's automatic inference of how 
const etc. is to be managed, just as design by contract is a doublecheck 
on the "contract" implied by a program's actual behavior.

-- Dai

On Thu, 01 May 2008 13:13:11 -0700, Davidson Corry wrote:

 Bruce Adams wrote:
 I agree with pretty much everything you've said. However, I'm not clear
 what point you're trying to make.


 I often feel that way myself. <grin>
 
 My point was that the language system ought to manage constness,
 invariance, synchronization on objects shared by multiple threads or
 processes, and so on, AUTOMATICALLY without requiring the programmer to
 manage those things explicitly (via keywords or whatever else), just as
 GC-enabled language systems now automatically manage memory allocation
 without requiring the programmer to use keywords etc. to do memory
 allocation explicitly.
 
 And my counterpoint was to suggest that, if the language system DOES
 provide keywords/properties/libraries by which const etc. CAN be managed
 explicitly if desired (but not required), then those tools should be an
 orthogonal doublecheck on the language's automatic inference of how
 const etc. is to be managed, just as design by contract is a doublecheck
 on the "contract" implied by a program's actual behavior.
 
 -- Dai


If I understand your request correctly, the reasons for const... is to 
allow the programmer to define the rules for his variables, not because 
the compiler can't see when a variable doesn't change.

The short, the compiler gets to enforce const and invariant on the 
programmer.

That is to say the compiler could take written code and identify what is 
invariant and const, but if someone else extends this those won't be 
marked in that way because they have changed it. How is the compiler 
going to say, "don't change that." If it has to guess from usage?

 Jesse Phillips wrote:
 
 If I understand your request correctly...


Well, not "request". Not even "proposal", at this point. More like "topic 
for speculation".

 ... the reasons for const... is to
 allow the programmer to define the rules for his variables, not
 because the compiler can't see when a variable doesn't change.
 
 The short, the compiler gets to enforce const and invariant on the
 programmer.
 
 That is to say the compiler could take written code and identify what
 is invariant and const, but if someone else extends this those won't
 be marked in that way because they have changed it. How is the
 compiler going to say, "don't change that." If it has to guess from
 usage?


Yes, that is one of two (at least two) reasons for using const etc.: to state 
a "declared constness contract" against which the /inferred/ constness behavior 
of the program can be double-checked. If the two contracts conflict -- an 
object is declared non-writable, but the program tries to write it -- the 
language can, at compile or at run time, catch the conflict and inform the 
programmer. But to infer this at compile time requires that the compiler 
have access to /all/ the source code for the system, not just for one source 
file.

And that is the second way to use these keywords: as a substitute for
inference. 
Let the programmer declare the "constness contract" explicitly, and don't 
do code analysis to infer it. That way, you don't have to have access to 
anything more than a single source file. But you lose the double-check aspect 
of the language construct.

You also make the programmer do more work, and you introduce the possibility 
that the programmer will make mistakes when doing that work. For instance, 
if the programmer declares constness one way in one module, and a subtly 
different way in another module, and links the two object files together, 
they may not work as expected and the linker can't tell us why. Unless,
perhaps, 
the compiler embeds constness-contract metadata into each object file, and 
the linker contains routines to analyze metadata from all object files it 
links, coordinate them and report conflicts... 

Bear also in mind that I am suggesting the language do whole-system inference, 
not only for constness issues, but also for exception safety (infers "nothrow" 
or "throws_only ExceptionX, ExceptionY and ExceptionZ") and for
multiprogramming 
rendezvous (infers "synchronized" etc.), just as the language now does for 
memory allocation (automatic garbage collection).

If the kind of whole-system inference I am suggesting is possible (and I 
don't know whether or not it is), then a programmer /wouldn't have to/ use 
keywords to declare explicit contracts for constness, exception safety,
multiprogramming 
etc. The compiler would just figure out what was needed and optimize for it.

And if the language does provide such keywords, and the programmer chooses 
to use them, then the compiler could do its analysis and say, "Well, you've 
told me that the program is going to do /this/ and /this/ and /this/. And 
I can see from my analysis that, no, it ain't. Let's talk."

-- Dai

Davidson Corry wrote:


If the kind of whole-system inference I am suggesting is possible (and I 
don't know whether or not it is), then a programmer /wouldn't have to/ use 
keywords to declare explicit contracts for constness, exception safety, 
multiprogramming etc. The compiler would just figure out what was needed 
and optimize for it.


Haskell attempts this. And does a pretty good job of it too. But at what 
cost?

A binary install occupies nearly 70 MB--it's hard to isolate how much of 
that is stricty necessary for its operation.

The last time I built it from sources, it took nearly 40 hours to compile.

But, most damning of all, is the extraordinary loops the programmer has to 
go through to do anything that involves processing large volumes of data 
efficiently. The purity ethic, ie. referential transparency, means that 
any time you want to modify a piece of information, you have to swap it 
for a newly allocated piece of memory holding the transformed entity and 
discard the old one as you pipeline them from one function to another.

Its a bit like a car assembly line. Except that every time you add a new 
component, you have to transfer what you've built so far to a new bogie. 
But of course you don't have an unlimited supply of bogies (memory) so you 
have to constantly recycle the bogies back to the beginning of the line 
and pick up a new one. Every step of the way, Bogie Control (BC) has to 
supply you with a new bogie and take back a (slightly) used one. Half of 
your total effort is spent in recycling bogies instead of building cars.

Yes. Haskell does have (unsafe) arrays, and you can use them to modify 
data in-place, to avoid the GC overhead. To use them, you have to give up 
most of the (supposed) benefits of purity, and revert to impure, 
procedural pseudo-C in Haskell. But to do that, you have to wrap 
everything up in a wrappers (Monads) to protect the precious purity from 
pollution. And in large part, you've discarded much of the reason for 
having the purity in the first place.

Before you get caught up in the hype of purity and the promises of 
lock-free, transparent multi-processing, ask to see a real world example 
of processing large amounts of data. It either doesn't exist; runs very 
slowly; or reverts to procedural techniques and non-referentially 
transparent data storage to achieve efficiency. And in the process forces 
the use of clumsy and inelegant syntax to do it.

There is a lesson to be learnt here. And it goes right back to the 
original mission statement for D's design philosophy. In a word: pragmatism.

Cheers, b.
-- 

Me Here wrote:
 There is a lesson to be learnt here. And it goes right back to the 
 original mission statement for D's design philosophy. In a word: 
 pragmatism.


I think your post makes it pretty clear that a pure functional, or pure 
inference, or pure anything language means it works well only if the 
particular programming problem falls 100% into that domain. It's like 
trying to do math when the only operation you have available is '+'.

The idea in D is that when your program needs to switch gears, it can do 
so while remaining in the same language, and without horrible kludges. 
This means that D will support functional, procedural, safe, unsafe, 
OOP, imperative, metaprogramming, etc., styles fairly seamlessly under 
one umbrella.

The downside is the language can become a bit kitchen-sinkish, but the 
upside is much larger. Look at the awful lengths people go to to mix 
Python and C++, for example. Or Erlang and C. Or Java and C++.

Languages that hew to the purity of one particular style tend to spawn 
evangelists and apologists for the One True Way who tie themselves into 
knots trying to prove that, for example, OOP fits every problem.

While C++ merges OOP and procedural (thereby deriving a huge benefit), I 
don't know of a language that merges functional with procedural. D is 
certainly going to give it a try.

 Davidson Corry wrote:

 If the kind of whole-system inference I am suggesting is possible
 (and I don't know whether or not it is), then a programmer /wouldn't
 have to/ use keywords to declare explicit contracts for constness,
 exception safety, multiprogramming etc. The compiler would just
 figure out what was needed and optimize for it.
 




 Me Here wrote:
 
 Haskell attempts this. And does a pretty good job of it too. But at
 what cost?
 
 [ ... discussions of Haskell inefficiencies deleted ... ]
 
 Before you get caught up in the hype of purity and the promises of
 lock-free, transparent multi-processing, ask to see a real world
 example of processing large amounts of data. It either doesn't exist;
 runs very slowly; or reverts to procedural techniques and
 non-referentially transparent data storage to achieve efficiency. And
 in the process forces the use of clumsy and inelegant syntax to do it.
 
 There is a lesson to be learnt here. And it goes right back to the
 original mission statement for D's design philosophy. In a word:
 pragmatism.


 Walter Bright wrote:

 ...The idea in D is that when your program needs to switch gears, it can
 do so while remaining in the same language, and without horrible
 kludges. This means that D will support functional, procedural, safe,
 unsafe, OOP, imperative, metaprogramming, etc., styles fairly
 seamlessly under one umbrella....
 I don't know of a language that merges functional with procedural. 
 D is certainly going to give it a try.


Well, and you make me think. 

What I had suggested was that the compiler might do whole-system analysis 
of the code, infer from that a contract for constness, synchronization etc., 
and then double-check that behavior against the other contract explicitly 
declared by the programmer. Perhaps we can turn that on its head: have an 
analysis system, maybe running in a background thread of a smart editor while 
you are actually keying in the program, analyze your code and suggest "this 
looks like it's const, and that function appears to be nothrow, and if you're 
targeting a multi-threaded platform you will probably want to synchronize 
this section of code here -- would you like me to add those annotations? 
Oh, and by the way, this new interface you're creating today has become
conflicted 
with the contract of that other module you linked to it yesterday." It doesn't 
have to be part of the compiler.

I am no fan of ivory-tower abstractions, particularly not at the cost of 
performance. I will happily mix paradigms to get the job done. (Heck, I love 
Perl. Q.E.D.)

I just want the computer to do the grunt work for me.

-- Dai

Walter Bright wrote:
 The idea in D is that when your program needs to switch gears, it can do 
 so while remaining in the same language, and without horrible kludges. 
 This means that D will support functional, procedural, safe, unsafe, 
 OOP, imperative, metaprogramming, etc., styles fairly seamlessly under 
 one umbrella.


If I had seen somebody write the above on some random webpage, I'd 
dismiss him out of hand as a kook.  But D is actually looking like it 
might reach that goal!

So I asked myself: are there any other common programming models that D 
doesn't have?  And it struck me: most interpreted languages have duck 
typing and hold-any-value variables.  Is there space for that in D?

Janice Caron wrote:
 On 03/05/2008, Russell Lewis <webmaster villagersonline.com> wrote:
  So I asked myself: are there any other common programming models that D
 doesn't have?  And it struck me: most interpreted languages have duck typing
 and hold-any-value variables.  Is there space for that in D?


 We already have hold-any-value variables!
 
 std.variant.Variant


Hmm...I'll have to go look that up!

 We also effectively have compile-time duck typing (but not runtime
 duck typing), because that's just how templates work. Runtime...? Hmm.
 That would require runtime reflection, I think.


After I wrote my previous post, it occurred to me the opDot, combined 
with an associative array, might give us duck typing.

In other words, we might *already* have all of this!  (Just hidden 
behind a little bit of syntax.)

On 03/05/2008, Russell Lewis <webmaster villagersonline.com> wrote:
  So I asked myself: are there any other common programming models that D
 doesn't have?  And it struck me: most interpreted languages have duck typing
 and hold-any-value variables.  Is there space for that in D?


We already have hold-any-value variables!

std.variant.Variant

We also effectively have compile-time duck typing (but not runtime
duck typing), because that's just how templates work. Runtime...? Hmm.
That would require runtime reflection, I think.