www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DMD 0.142 release

reply "Walter Bright" <newshound digitalmars.com> writes:
This incorporates a new 'header' generator capability, written by Dave
Fladebo, now working!

http://www.digitalmars.com/d/changelog.html
Dec 29 2005
next sibling parent reply Deewiant <deewiant.doesnotlike.spam gmail.com> writes:
Walter Bright wrote:
 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!
 

Thanks for the release, but "partly" is missing from before "working" (as usual when it comes to programming <g>). I presume bugs related to the header generator still belong in digitalmars.D.bugs? I'll post my report here anyway since it's so short. The following crashes DMD completely when compiled with -H, on Windows (XP) at least: this() {} In or outside a class, doesn't matter, it still doesn't work.
Dec 29 2005
parent reply Deewiant <deewiant.doesnotlike.spam gmail.com> writes:
Deewiant wrote:
 this() {}
 
 In or outside a class, doesn't matter, it still doesn't work.

I should probably correct myself somewhat: if it's inside a class the header file is correctly generated, but DMD still crashes.
Dec 29 2005
parent "Walter Bright" <newshound digitalmars.com> writes:
"Deewiant" <deewiant.doesnotlike.spam gmail.com> wrote in message 
news:dp1epm$nta$1 digitaldaemon.com...
 Deewiant wrote:
 this() {}

 In or outside a class, doesn't matter, it still doesn't work.

I should probably correct myself somewhat: if it's inside a class the header file is correctly generated, but DMD still crashes.

Thanks, I'll take care of it.
Dec 29 2005
prev sibling next sibling parent reply "Derek Parnell" <derek psych.ward> writes:
On Fri, 30 Dec 2005 06:11:54 +1100, Walter Bright  
<newshound digitalmars.com> wrote:

 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!

Well it might be working, but it doesn't do what I was expecting. This source file ... //-------------------------------- import std.stdio; private int qwerty; class A { private int x; void ths() { x = 1; } } void xpub() { A a = new A; qwerty = 1; } private void xpriv() { A b = new A; qwerty = 2; } //-------------------------------- was turned into this 'header' ... // D import file generated from 'test.d' import std.stdio; private { int qwerty; } class A { private { int x; } void ths() { x = 1; } } void xpub() { A a = new A; qwerty = 1; } private { void xpriv() { A b = new A; qwerty = 2; } } //------------------------- but I was expecting something more like this ... // D import file generated by hand import std.stdio; class A { void ths(){} } void xpub(){} //------------------------- In other words, why does the generated header show implementation code and why does it show private members? Currently, all it seems to do is reformat the original source by placing braces around private sections. -- Derek Parnell Melbourne, Australia
Dec 29 2005
next sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Derek Parnell" <derek psych.ward> wrote in message 
news:op.s2kab6y06b8z09 ginger.vic.bigpond.net.au...
 In other words, why does the generated header show implementation code and 
 why does it show private members?

It shows implementation code for functions that are candidates for inlining. It shows private members because they might be used in const initializers and in inline functions. If a function cannot be inlined, its implementation will not be included.
Dec 29 2005
next sibling parent reply John Reimer <terminal.node gmail.com> writes:
Walter Bright wrote:
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:op.s2kab6y06b8z09 ginger.vic.bigpond.net.au...
 In other words, why does the generated header show implementation code and 
 why does it show private members?

It shows implementation code for functions that are candidates for inlining. It shows private members because they might be used in const initializers and in inline functions. If a function cannot be inlined, its implementation will not be included.

This really doesn't look right. I realize the need to make private data visible for inlining purposes, but this appears to defeat the main purpose of the interface file -- hiding private members and implementation details. The need to accommodate inlining seems wrong too. Don't you think it's better, at this point, to disable inlining for interface files and document it well so nobody is surprised by the absence of it. Maybe, later on, a better solution can be concocted. -JJR
Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"John Reimer" <terminal.node gmail.com> wrote in message 
news:dp1ja1$rh2$1 digitaldaemon.com...
 I realize the need to make private data visible for inlining purposes, but 
 this appears to defeat the main purpose of the interface file -- hiding 
 private members and implementation details.

 The need to accommodate inlining seems wrong too.  Don't you think it's 
 better, at this point, to disable inlining for interface files and 
 document it well so nobody is surprised by the absence of it.

This would be a perfectly valid solution, but Dave and I decided to come at it from the direction of having the results be identical whether importing a .d or a .di file. Note that template bodies also wind up being totally reproduced in a .di file. Private data members are also needed in order to get the class member offsets right. In this way, it's like a C++ header file, or a Java .class file. I could switch it to a binary format, and that would provide some level of obfuscation, but that shouldn't be confused with a secure format. Binary symbol formats are easilly hacked and reversed. Even if the file format was encrypted, the decryption keys would have to be supplied to the compiler, and once DVD Jon reverse engineers it, that's the end of any security. The best way to achieve hiding is to use interface classes, not an interface file. The main advantage of a D interface file is compile performance. (P.S. binary symbol table formats have been compared to object files as far as difficulty of reverse engineering. I believe this is a misguided and completely false analogy. Reverse engineering an object file is converting the proverbial hamburger back into a cow, because most semantic information is thrown away in the process of making an object file. Not so with a symbol file; all the semantic information *must* be present. This thinking is the result of my discussions with C++ experts on having binary exported template symbol files, which many otherwise well informed people mistakenly imagine will make for a securely hidden implementation. ) (P.P.S. A D interface file can also be compared with a Java .class file, which can be imported by Java compilers rather than source code. But many tools exist which automatically turn .class files into reasonable Java source code. .class files don't really hide information at all.)
Dec 29 2005
parent reply Derek Parnell <derek psych.ward> writes:
On Thu, 29 Dec 2005 13:38:48 -0800, Walter Bright wrote:

 The main advantage of a D interface file is compile performance.

This is the *only* advantage and that's not such a problem anyhow. It seems then that I won't be needing the -H switch anytime soon. Excuse me for sounding impertinent and selfish, but are you spending your time on D wisely? We get something of little value and still not receive things that are higher on the priority list. How can I help? I know I can /donate/ library code and documentation, but the core language is still where the pre-1.0 issues lie. And even when I have given you library fixes, they are still to appear. I guess I don't really understand your organization's development processes. -- Derek (skype: derek.j.parnell) Melbourne, Australia "A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.'" - D.N. Adams 30/12/2005 9:41:09 AM
Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Derek Parnell" <derek psych.ward> wrote in message 
news:1uxk73kt2jabq.1w7j11urctxbb.dlg 40tude.net...
 On Thu, 29 Dec 2005 13:38:48 -0800, Walter Bright wrote:
 The main advantage of a D interface file is compile performance.


No, it does wind up hiding the implementation details of most functions.
 and that's not such a problem anyhow.

Build speed isn't a problem now, but it will become one once D projects exceed a certain size.
 It seems then that I won't be needing the -H switch anytime soon.

It's certainly a forward looking capability.
 Excuse me for sounding impertinent and selfish, but are you spending your
 time on D wisely? We get something of little value and still not receive
 things that are higher on the priority list.

It is regularly asked for. The specific impetus in this case was that Dave Fladebo went ahead and implemented it.
 How can I help? I know I can /donate/ library code and documentation, but
 the core language is still where the pre-1.0 issues lie. And even when I
 have given you library fixes, they are still to appear. I guess I don't
 really understand your organization's development processes.

It's mainly driven by 1) what people feel like contributing and 2) things that are blocking use of D.
Dec 29 2005
next sibling parent Chris Lajoie <ctlajoie___remove___this___ ___gmail.com> writes:
Walter Bright wrote:
 How can I help? I know I can /donate/ library code and documentation, but
 the core language is still where the pre-1.0 issues lie. And even when I
 have given you library fixes, they are still to appear. I guess I don't
 really understand your organization's development processes.

It's mainly driven by 1) what people feel like contributing and 2) things that are blocking use of D.

I believe Derek is asking how he can help the development of D. There is no indication that many of the things that need to be implemented/fixed are even being considered yet. Also, I have seen complaints from a few people now that their fixes aren't making it into phobos. Chris
Dec 30 2005
prev sibling parent reply "Dave" <Dave_member pathlink.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote in message
news:dp1s4k$12ev$2 digitaldaemon.com...
 "Derek Parnell" <derek psych.ward> wrote in message
 news:1uxk73kt2jabq.1w7j11urctxbb.dlg 40tude.net...
 On Thu, 29 Dec 2005 13:38:48 -0800, Walter Bright wrote:
 The main advantage of a D interface file is compile performance.


No, it does wind up hiding the implementation details of most functions.
 and that's not such a problem anyhow.

Build speed isn't a problem now, but it will become one once D projects exceed a certain size.
 It seems then that I won't be needing the -H switch anytime soon.

It's certainly a forward looking capability.
 Excuse me for sounding impertinent and selfish, but are you spending your
 time on D wisely? We get something of little value and still not receive
 things that are higher on the priority list.


Maybe it isn't a big step, at least at this stage in D's lifetime, but if so it may have cost me a bunch of time, not Walter (AFAIK). He integrated (and improved) this over the Holiday season and I reckon he deserves a break once in a while <g> It looks like there may be a few issues right now, but once -H is stable, build the phobos and DWT .di's and then compare and perhaps you'll see more value: 1) lots less code for the compiler to cut through and 2) the inlined functions really don't give all that much away, IMO. During my testing, for phobos the code reduction was ~50% and for DWT ~80%. Not to mention that skipping the semantic analysis for large function bodies should save a good chunk of compiler time. Once I saw how DWT was put together, one of the things I thought this would be useful for would be things like RAD IDE's. RAD IDE's should be a lot easier to build if you can import a couple of 'master headers' that then import the rest. If you can reduce the avg. 'header' by 80% for a GUI library (like DWT), the edit-compile-debug cycles should be much shorter, plus you simplify the IDE with respect to maintaining imports and the like.
 It is regularly asked for. The specific impetus in this case was that Dave
 Fladebo went ahead and implemented it.

The -H stuff was built using dmdfe(*) and then I wrote Walter regarding what it did and how it was tested to see if it made sense to include in the compiler before posting it as a seperate tool. It was intentionally built to be "easy" to plug-in to the dmd front end, but I also found out that it pretty much needs a full-blown compiler to do things right. The original dmdfe implementation was tested by producing 'headers' for phobos, DWT and several other test cases covering all of the examples in the documentation. I experimented with the "implementation hiding" details previously mentioned in other posts, but that was abandoned for the reasons also previously mentioned in other posts (see: digitalmars.D/29883). * dmdfe: http://home.comcast.net/~benhinkle/dmdfe/
 How can I help? I know I can /donate/ library code and documentation, but
 the core language is still where the pre-1.0 issues lie. And even when I
 have given you library fixes, they are still to appear. I guess I don't
 really understand your organization's development processes.

It's mainly driven by 1) what people feel like contributing and 2) things that are blocking use of D.

Dec 30 2005
parent reply John Reimer <terminal.node gmail.com> writes:
 Maybe it isn't a big step, at least at this stage in D's lifetime, but if so 
 it may have cost me a bunch of time, not Walter (AFAIK). He integrated (and 
 improved) this over the Holiday season and I reckon he deserves a break once 
 in a while <g>
 
 It looks like there may be a few issues right now, but once -H is stable, 
 build the phobos and DWT .di's and then compare and perhaps you'll see more 
 value: 1) lots less code for the compiler to cut through and 2) the inlined 
 functions really don't give all that much away, IMO.
 
 During my testing, for phobos the code reduction was ~50% and for DWT ~80%. 
 Not to mention that skipping the semantic analysis for large function bodies 
 should save a good chunk of compiler time.
 
 Once I saw how DWT was put together, one of the things I thought this would 
 be useful for would be things like RAD IDE's. RAD IDE's should be a lot 
 easier to build if you can import a couple of 'master headers' that then 
 import the rest. If you can reduce the avg. 'header' by 80% for a GUI 
 library (like DWT), the edit-compile-debug cycles should be much shorter, 
 plus you simplify the IDE with respect to maintaining imports and the like.
 
 It is regularly asked for. The specific impetus in this case was that Dave
 Fladebo went ahead and implemented it.

The -H stuff was built using dmdfe(*) and then I wrote Walter regarding what it did and how it was tested to see if it made sense to include in the compiler before posting it as a seperate tool. It was intentionally built to be "easy" to plug-in to the dmd front end, but I also found out that it pretty much needs a full-blown compiler to do things right. The original dmdfe implementation was tested by producing 'headers' for phobos, DWT and several other test cases covering all of the examples in the documentation. I experimented with the "implementation hiding" details previously mentioned in other posts, but that was abandoned for the reasons also previously mentioned in other posts (see: digitalmars.D/29883). * dmdfe: http://home.comcast.net/~benhinkle/dmdfe/

Walter is not always the best at "selling" these things to us. You've done a lot better job here. :) Still, I wonder how these headers look for DWT, which I'm pretty sure has lots of opportunities for inlining and private data. -JJR
Dec 30 2005
parent "Dave" <Dave_member pathlink.com> writes:
"John Reimer" <terminal.node gmail.com> wrote in message 
news:dp4lhd$4m1$1 digitaldaemon.com...

 During my testing, for phobos the code reduction was ~50% and for DWT 
 ~80%.


 Still, I wonder how these headers look for DWT, which I'm pretty sure has 
 lots of opportunities for inlining and private data.

I just re-checked my numbers and it looks to be >= 80% ;) But, it is pretty hard to do really accurately with a quick grep -v | wc type of thing because of different whitespace delimiters, etc. To do that really well you'd almost need to add it to the compiler.. hmmm... <JUST KIDDING <g>> What I've noticed is that there are fewer opportunities for inlining then you might think in DWT and in general for anything non-trivial... For example, any fbody with a loop in it is not a candidate, which generally is a real good thing, I think. OTOH, Phobos std.math[2] has just about everything in it inlined, which is also a good thing, since most of those are return statements basically codifying what the name of the function implies. There was a great article a while back in Dr. Dobbs with an inlining algorithm that made a lot of sense, and I think the DMD frontend heuristics and the article coincide quite a bit (for the record, DMD had implemented the inlining long before the article came out ;). The reason I mention this is because I think compiler developers are starting to nail down what should be inlined pretty well, so the inlining is good to have in the reference compiler and reasonable to have in the -H tool too.
 -JJR

Dec 31 2005
prev sibling next sibling parent "Derek Parnell" <derek psych.ward> writes:
On Fri, 30 Dec 2005 07:40:24 +1100, Walter Bright  
<newshound digitalmars.com> wrote:

 "Derek Parnell" <derek psych.ward> wrote in message
 news:op.s2kab6y06b8z09 ginger.vic.bigpond.net.au...
 In other words, why does the generated header show implementation code  
 and
 why does it show private members?

It shows implementation code for functions that are candidates for inlining.

Thanks for explaining this. I don't like it one bit, though. The DI file can be used instead of the original source when the implementation is in a library right? How does the compiler know if something has been inlined or not in the library object? Just because a specific edition of DMD reckons that a function is a candidate for inlining, it cannot know that some other edition of some other D compiler has actually inlined it. Also I thought that a DI file is not being used to generate object code, but just to get information about how to compile the other (non) DI fi
 It shows private members because they might be used in const initializers
 and in inline functions.

But 'private' means that I don't want other modules knowing about this.
 If a function cannot be inlined, its implementation will not be included.

So we must make functions we want to hide deliberately complex to prevent inlining? This just sounds wrong. Isn't there a way to turn off inlining if I don't want it? I wanted to use DI files to *HIDE* implementation details. Your definition of DI does not do this. -- Derek Parnell Melbourne, Australia
Dec 29 2005
prev sibling parent reply "Chris Miller" <chris dprogramming.com> writes:
On Thu, 29 Dec 2005 15:40:24 -0500, Walter Bright  
<newshound digitalmars.com> wrote:

 "Derek Parnell" <derek psych.ward> wrote in message
 news:op.s2kab6y06b8z09 ginger.vic.bigpond.net.au...
 In other words, why does the generated header show implementation code  
 and
 why does it show private members?

It shows implementation code for functions that are candidates for inlining. It shows private members because they might be used in const initializers and in inline functions. If a function cannot be inlined, its implementation will not be included.

Perhaps it should only do this if -inline is present.
Dec 29 2005
parent reply John Reimer <terminal.node gmail.com> writes:
Chris Miller wrote:
 On Thu, 29 Dec 2005 15:40:24 -0500, Walter Bright 
 <newshound digitalmars.com> wrote:
 
 "Derek Parnell" <derek psych.ward> wrote in message
 news:op.s2kab6y06b8z09 ginger.vic.bigpond.net.au...
 In other words, why does the generated header show implementation 
 code and
 why does it show private members?

It shows implementation code for functions that are candidates for inlining. It shows private members because they might be used in const initializers and in inline functions. If a function cannot be inlined, its implementation will not be included.

Perhaps it should only do this if -inline is present.

What happens, then, if you use "-inline" on the library and no "-inline" on the project that uses the library? You get an interface file that's prepared for inline code per the library build (with all the extra private and function implementation details), but a project file that uses an interface file that won't be inlined in the project. Kind of complicated. I think it's just better to disable inlining on libraries that generate an interface file and on projects that uses that interface file. -JJR
Dec 29 2005
parent John Reimer <terminal.node gmail.com> writes:
John Reimer wrote:
 Chris Miller wrote:
 On Thu, 29 Dec 2005 15:40:24 -0500, Walter Bright 
 <newshound digitalmars.com> wrote:

 "Derek Parnell" <derek psych.ward> wrote in message
 news:op.s2kab6y06b8z09 ginger.vic.bigpond.net.au...
 In other words, why does the generated header show implementation 
 code and
 why does it show private members?

It shows implementation code for functions that are candidates for inlining. It shows private members because they might be used in const initializers and in inline functions. If a function cannot be inlined, its implementation will not be included.

Perhaps it should only do this if -inline is present.

What happens, then, if you use "-inline" on the library and no "-inline" on the project that uses the library? You get an interface file that's prepared for inline code per the library build (with all the extra private and function implementation details), but a project file that uses an interface file that won't be inlined in the project. Kind of complicated. I think it's just better to disable inlining on libraries that generate an interface file and on projects that uses that interface file. -JJR

I should qualify the above. "-inline" should be allowed on the project file as usual, but only for items that don't pertain to the interface file.
Dec 29 2005
prev sibling next sibling parent reply "Kris" <fu bar.com> writes:
Regarding Derek's example: what exactly is the value in an "import file" 
like that, Walter? It bears little or no resemblance to what any related 
posts over the last two years have been requesting, and seems to break all 
understood rules vis-a-vis visability.

It certainly appears to have little bearing on the primary request point ~ 
that is, taking a non OSS component and exposing the minimal attributes to 
make it usable/linkable. The example below explicitly exposes the inner 
workings (private attributes also!), which is totally contrary to said 
requested purpose.

I really don't get the point of this at all, and it seems like others are in 
the same boat. Can you enlighten us, please? It's not April 1st yet, right? 
:)

- Kris


"Derek Parnell" <derek psych.ward> wrote in message 
news:op.s2kab6y06b8z09 ginger.vic.bigpond.net.au...
 On Fri, 30 Dec 2005 06:11:54 +1100, Walter Bright 
 <newshound digitalmars.com> wrote:

 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!

Well it might be working, but it doesn't do what I was expecting. This source file ... //-------------------------------- import std.stdio; private int qwerty; class A { private int x; void ths() { x = 1; } } void xpub() { A a = new A; qwerty = 1; } private void xpriv() { A b = new A; qwerty = 2; } //-------------------------------- was turned into this 'header' ... // D import file generated from 'test.d' import std.stdio; private { int qwerty; } class A { private { int x; } void ths() { x = 1; } } void xpub() { A a = new A; qwerty = 1; } private { void xpriv() { A b = new A; qwerty = 2; } } //------------------------- but I was expecting something more like this ... // D import file generated by hand import std.stdio; class A { void ths(){} } void xpub(){} //------------------------- In other words, why does the generated header show implementation code and why does it show private members? Currently, all it seems to do is reformat the original source by placing braces around private sections. -- Derek Parnell Melbourne, Australia

Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp1on5$vsn$1 digitaldaemon.com...
 I really don't get the point of this at all, and it seems like others are 
 in the same boat. Can you enlighten us, please? It's not April 1st yet, 
 right? :)

Sure. See my other post in this thread.
Dec 29 2005
parent reply "Kris" <fu bar.com> writes:
You suggest that using Interfaces is the right way to hide implementation 
detail. I'd agree, but how does one expose the means of accessing an 
instance? Does one provide an interface factory, and expose a header for 
that?

How is it supposed to work?


"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dp1pbv$10i7$2 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dp1on5$vsn$1 digitaldaemon.com...
 I really don't get the point of this at all, and it seems like others are 
 in the same boat. Can you enlighten us, please? It's not April 1st yet, 
 right? :)

Sure. See my other post in this thread.

Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp1sn8$12rr$1 digitaldaemon.com...
 You suggest that using Interfaces is the right way to hide implementation 
 detail. I'd agree, but how does one expose the means of accessing an 
 instance? Does one provide an interface factory, and expose a header for 
 that?

That's the classic way of doing it.
Dec 29 2005
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Thu, 29 Dec 2005 16:03:22 -0800, Walter Bright wrote:

 "Kris" <fu bar.com> wrote in message news:dp1sn8$12rr$1 digitaldaemon.com...
 You suggest that using Interfaces is the right way to hide implementation 
 detail. I'd agree, but how does one expose the means of accessing an 
 instance? Does one provide an interface factory, and expose a header for 
 that?

That's the classic way of doing it.

Can someone translate this into an example or two? -- Derek (skype: derek.j.parnell) Melbourne, Australia "A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.'" - D.N. Adams 30/12/2005 11:18:52 AM
Dec 29 2005
parent reply "Kris" <fu bar.com> writes:
Here's an arbitrary example, Derek ~ for a closed-source SearchEngine. I 
think Walter is the only one who can explain how -H can help here:

~~~~~~~~~~~~~~~~~~~
module SearchFactory;

// these interfaces would typically live in other modules

// the hidden implementation
interface ISearchEngine
{
        ISearchItem[] lookup (char[] terms);
}

// more hidden implementation
interface ISearchItem
{
        int relevancy();
        char[] retrievalUrl();
        char[] displaySnippet();
}


// import a.bunch.of.proprietary.stuff;


// a factory for hooking up a client to the hidden implementation
ISearchEngine createSearchEngine (char[] someAttributes)
{
        // potentially use a.bunch.of.proprietary.stuff;
}


// do some proprietary setup
static this()
{
        // potentially use a.bunch.of.proprietary.stuff;
}

~~~~~~~~~~~~~~~~~~~~~

unfortunately, this is what I get when using the -H option:
dmd.exe has encountered a problem and needs to close. We are sorry for the 
inconvenience.


GPF's aside, I imagine that -H would expose the content of both the 
createSearchEngine() factory method, and whatever else is in there (such as 
the static ctor). This means that the all relevant proprietary import 
modules will also need to be shipped with this module :-)

The idea of an Interface is to cleanly isolate a "contractual obligation" 
from any and all implementation detail. I could easily be wrong, but I fail 
to see how this can be achieved with D (even in a trivial example) without 
rather careful attention to detail.

If I were to create a "di" file by hand, then it might look something like 
this:

~~~~~~~~~~~~~~~~~~~~~~~
module importSearchEngine;

// expose the interfaces
public import search.ISearchEngine;
public import search.ISearchItem;

ISearchEngine createSearchEngine (char[] someAttributes);
~~~~~~~~~~~~~~~~~~~~~~~~

This should work correctly (note the lack of implementation detail). 
However, it bears little or no resemblance to the -H option. Thus -H is 
worthless for such things ~ I think you have to build them by hand instead 
...

:-(





"Derek Parnell" <derek psych.ward> wrote in message 
news:bm7jarnr2vgq$.ddyfy4fndls1.dlg 40tude.net...
 On Thu, 29 Dec 2005 16:03:22 -0800, Walter Bright wrote:

 "Kris" <fu bar.com> wrote in message 
 news:dp1sn8$12rr$1 digitaldaemon.com...
 You suggest that using Interfaces is the right way to hide 
 implementation
 detail. I'd agree, but how does one expose the means of accessing an
 instance? Does one provide an interface factory, and expose a header for
 that?

That's the classic way of doing it.

Can someone translate this into an example or two? -- Derek (skype: derek.j.parnell) Melbourne, Australia "A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.'" - D.N. Adams 30/12/2005 11:18:52 AM

Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp20nn$15ns$1 digitaldaemon.com...
 Here's an arbitrary example, Derek ~ for a closed-source SearchEngine. I 
 think Walter is the only one who can explain how -H can help here:

 ~~~~~~~~~~~~~~~~~~~
 module SearchFactory;

 // these interfaces would typically live in other modules

 // the hidden implementation
 interface ISearchEngine
 {
        ISearchItem[] lookup (char[] terms);
 }

 // more hidden implementation
 interface ISearchItem
 {
        int relevancy();
        char[] retrievalUrl();
        char[] displaySnippet();
 }


 // import a.bunch.of.proprietary.stuff;


 // a factory for hooking up a client to the hidden implementation
 ISearchEngine createSearchEngine (char[] someAttributes)
 {
        // potentially use a.bunch.of.proprietary.stuff;
 }


 // do some proprietary setup
 static this()
 {
        // potentially use a.bunch.of.proprietary.stuff;
 }

 ~~~~~~~~~~~~~~~~~~~~~

 unfortunately, this is what I get when using the -H option:
 dmd.exe has encountered a problem and needs to close. We are sorry for the 
 inconvenience.

I'd like an example of that, so I can fix it.
 GPF's aside, I imagine that -H would expose the content of both the 
 createSearchEngine() factory method, and whatever else is in there (such 
 as the static ctor). This means that the all relevant proprietary import 
 modules will also need to be shipped with this module :-)

No, it won't if the function is too complex to be inlined, which is the usual case.
 The idea of an Interface is to cleanly isolate a "contractual obligation" 
 from any and all implementation detail. I could easily be wrong, but I 
 fail to see how this can be achieved with D (even in a trivial example) 
 without rather careful attention to detail.

A good example of using interfaces is COM programming under Windows. The general idea is: interface Foo { T method1(args); U method2(args); } Foo FooFactory(args); and you call FooFactory(args) to create an instance of Foo, instead of using new.
Dec 29 2005
next sibling parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote
[snip]
 unfortunately, this is what I get when using the -H option:
 dmd.exe has encountered a problem and needs to close. We are sorry for 
 the inconvenience.

I'd like an example of that, so I can fix it.

Uh, the very example was included right above your comment?
 GPF's aside, I imagine that -H would expose the content of both the 
 createSearchEngine() factory method, and whatever else is in there (such 
 as the static ctor). This means that the all relevant proprietary import 
 modules will also need to be shipped with this module :-)

No, it won't if the function is too complex to be inlined, which is the usual case.

I see. But, surely that makes it non-deterministic in a variety of manners? Thus, one could not always depend on such a header being generated correctly?
Dec 29 2005
parent "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp27jl$19bc$1 digitaldaemon.com...
 "Walter Bright" <newshound digitalmars.com> wrote
 [snip]
 unfortunately, this is what I get when using the -H option:
 dmd.exe has encountered a problem and needs to close. We are sorry for 
 the inconvenience.

I'd like an example of that, so I can fix it.

Uh, the very example was included right above your comment?

Ok. I didn't realize that. Thanks!
 GPF's aside, I imagine that -H would expose the content of both the 
 createSearchEngine() factory method, and whatever else is in there (such 
 as the static ctor). This means that the all relevant proprietary import 
 modules will also need to be shipped with this module :-)

No, it won't if the function is too complex to be inlined, which is the usual case.

manners? Thus, one could not always depend on such a header being generated correctly?

The code and header are still correct whether it is inlined or not.
Dec 29 2005
prev sibling next sibling parent Derek Parnell <derek psych.ward> writes:
On Thu, 29 Dec 2005 18:13:03 -0800, Walter Bright wrote:

 I'd like an example of that, so I can fix it.

module cf; interface iFoo{} class Foo: iFoo{} C:\> dmd -c -H cf.d ... ABEND ... -- Derek (skype: derek.j.parnell) Melbourne, Australia "A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.'" - D.N. Adams 30/12/2005 3:36:47 PM
Dec 29 2005
prev sibling parent reply Derek Parnell <derek psych.ward> writes:
On Thu, 29 Dec 2005 18:13:03 -0800, Walter Bright wrote:

 
 I'd like an example of that, so I can fix it.

module cf; class Foo { this(){} } ----------------- module cf; static this() { } -- Derek (skype: derek.j.parnell) Melbourne, Australia "A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.'" - D.N. Adams 30/12/2005 3:40:41 PM
Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
Thanks for the examples. I'll take care of it. 
Dec 29 2005
parent reply JT <JT_member pathlink.com> writes:
Thanks Walter this import file is a nice feature. :D

And I agree - the pretty printing is usefull to have in the front end. I noticed
while playing around in the front end that you have stubbed out the posibility
of using 'precompiled symbol files' as imports - do you still have plans for
this and do you expect it to be in version 1?
Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"JT" <JT_member pathlink.com> wrote in message 
news:dp2ins$1gn3$1 digitaldaemon.com...
 And I agree - the pretty printing is usefull to have in the front end. I 
 noticed
 while playing around in the front end that you have stubbed out the 
 posibility
 of using 'precompiled symbol files' as imports - do you still have plans 
 for
 this and do you expect it to be in version 1?

.di files replace precompiled symbol files.
Dec 30 2005
parent JT <JT_member pathlink.com> writes:
In article <dp50o3$bt6$1 digitaldaemon.com>, Walter Bright says...
"JT" <JT_member pathlink.com> wrote in message 
news:dp2ins$1gn3$1 digitaldaemon.com...
 And I agree - the pretty printing is usefull to have in the front end. I 
 noticed
 while playing around in the front end that you have stubbed out the 
 posibility
 of using 'precompiled symbol files' as imports - do you still have plans 
 for
 this and do you expect it to be in version 1?

.di files replace precompiled symbol files.

very cool. .di files make much more sense now, in fact I think its a very good solution for this part. Thanks for everything you do Walter.
Dec 30 2005
prev sibling parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote
 "Kris" <fu bar.com> wrote in message 
 news:dp1sn8$12rr$1 digitaldaemon.com...
 You suggest that using Interfaces is the right way to hide implementation 
 detail. I'd agree, but how does one expose the means of accessing an 
 instance? Does one provide an interface factory, and expose a header for 
 that?

That's the classic way of doing it.

Then, this related "di" would not expose anything private or potentially "sensitive" about the interface-factory itself? Can you perhaps illustrate with a working example, please? I think that would help a lot :)
Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp1ufk$142q$1 digitaldaemon.com...
 "Walter Bright" <newshound digitalmars.com> wrote
 "Kris" <fu bar.com> wrote in message 
 news:dp1sn8$12rr$1 digitaldaemon.com...
 You suggest that using Interfaces is the right way to hide 
 implementation detail. I'd agree, but how does one expose the means of 
 accessing an instance? Does one provide an interface factory, and expose 
 a header for that?

That's the classic way of doing it.

Then, this related "di" would not expose anything private or potentially "sensitive" about the interface-factory itself? Can you perhaps illustrate with a working example, please? I think that would help a lot :)

.di files aren't necessarilly going to help you here. But in another post in this thread, I gave an example of FooFactory() that is the usual way of generating an instance of an interface.
Dec 29 2005
parent "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote
 "Kris" <fu bar.com> wrote
 Can you perhaps illustrate with a working example, please? I think that 
 would help a lot :)

.di files aren't necessarilly going to help you here. But in another post in this thread, I gave an example of FooFactory() that is the usual way of generating an instance of an interface.

As did I, which showed two things: a) -H is of little realistic value (if any) for exposing factories used for purposes of code hiding. In fact, it just gets in the way and should not be used for said purpose. b) -H currently fails if you try to do the above (example in the prior post). I think there's been a total miscommunication here. Seems like most everyone was expecting some form of code-hiding per the long-standing requests for closed-source support (certainly I was thoroughly misguided). Instead, what -H does is some form of source-munging that makes it questionably faster for the compiler to parse? I recall something from a long time back noting that just such a facility would be implemented via a binary format? As such, it would very likely be much faster than the current -H approach (which requires full lexing and parsing of the exposed source). Without wishing to offend anyone, it seems like what's been implemented will be unlikely to make a /blind bit of difference/ to compile times (in contrast to that binary-format version you had mentioned so long ago). Yes, it's great that you took an opportunity to clean up a bunch of output-formatting issues, but .... this new 'feature' appears to exist simply and only because it can? I guess one could split hairs over that, all day. What this feature /does/ do is give the user yet another darned file-type to bother about, and provides a non-deterministic means to reformat ones source-code. That just doesn't seem like such a great idea, somehow. Oh well.
Dec 29 2005
prev sibling parent reply Derek Parnell <derek psych.ward> writes:
On Fri, 30 Dec 2005 07:10:44 +1100, Derek Parnell wrote:

Ok, how about we try to get both camps happy about this DI file concept.
Here is a suggestion that does mean a change to the language though...
 
Given this source file ...
 
 //--------------------------------
 import std.stdio;
 
 private int  qwerty;
 private real asdfgh;

 pragma(generate): // Turn inlining off from here on.
 
 class A
 {
      private int x;
      void ths()
      {
        x = 1;
        }
 }
 
 void xpub()
 {
      A a = new A;
      qwerty = 1;
 }
 
 private void xpriv()
 {
      A b = new A;
      qwerty = 2;
 }
 //--------------------------------
 

It could be turned into this 'header' ...

 // D import file generated from 'test.d'
 import std.stdio;
 reserve:14; // fourteen bytes reserved
 class A
 {
  reserve:4; // four bytes reserved
  void ths() { }
 }
 void xpub() { }
 //-------------------------


In other words, let's enhance the language so we can support the needs of
developers who want closed-source development *and* the needs of the
compiler to generate correctly performing code. This would mean we need to
resolve the issue of class member offsets, inlining, and const
declarations. These are not impossible to resolve.

-- 
Derek
(skype: derek.j.parnell)
Melbourne, Australia
"A learning experience is one of those things that says,
 'You know that thing you just did? Don't do that.'" - D.N. Adams
30/12/2005 9:56:10 AM
Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Derek Parnell" <derek psych.ward> wrote in message 
news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 In other words, let's enhance the language so we can support the needs of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we need to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

There's more than that, there are template bodies. At some point, we have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.
Dec 29 2005
next sibling parent reply James Dunne <james.jdunne gmail.com> writes:
Walter Bright wrote:
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 
In other words, let's enhance the language so we can support the needs of
developers who want closed-source development *and* the needs of the
compiler to generate correctly performing code. This would mean we need to
resolve the issue of class member offsets, inlining, and const
declarations. These are not impossible to resolve.

There's more than that, there are template bodies. At some point, we have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

Understood. How about a separate interface class generation feature then? If it were up to me I would rename the 'import file' feature to 'precompiled header', hoping that the term is not copyrighted/trademarked by Microsoft. =) Then again, if they work 'a lot like' precompiled headers in C++ then what are the main differences if the two are not exactly alike? For the closed source community, one could also invent a specially formatted D import source file which declares only the non-private methods of classes *and* at which offsets they can be found at in the vtable. Of course include the public fields and at which offsets they are found at as well.
Dec 29 2005
next sibling parent "Walter Bright" <newshound digitalmars.com> writes:
"James Dunne" <james.jdunne gmail.com> wrote in message 
news:dp1t3o$1383$1 digitaldaemon.com...
 Walter Bright wrote:
 They work a lot like "precompiled headers" do in C++.

How about a separate interface class generation feature then? If it were up to me I would rename the 'import file' feature to 'precompiled header', hoping that the term is not copyrighted/trademarked by Microsoft. =) Then again, if they work 'a lot like' precompiled headers in C++ then what are the main differences if the two are not exactly alike?

The main difference is that precompiled headers are an abominable kludge <g>. Precompiled headers are order dependent, and cannot be included multiple times. Some things cannot be put in headers to be precompiled. Precompiled headers tend to aggregate all the headers together. They are very sensitive to compiler switch settings, meaning they need to be constantly rebuilt. None of these apply to D interface files.
 For the closed source community, one could also invent a specially 
 formatted D import source file which declares only the non-private methods 
 of classes *and* at which offsets they can be found at in the vtable.  Of 
 course include the public fields and at which offsets they are found at as 
 well.

Because of templates, const initializers, etc., I suspect that although this is doable, it will be of marginal utility.
Dec 29 2005
prev sibling parent reply BCS <BCS_member pathlink.com> writes:
Why dose the order of the function in a class define the order of there 
entries in the vtbl? This seems to me like an unnecessary complication 
that could create some vary difficult to find bugs.

Example:
Say an imported file gets changed but never gets recompiled (maybe it's 
a header file to a closed library)

1) compile main so that it imports new a.
2) compile old a
3) link

This won't have any errors but will end up calling the wrong fn at run time.
----------------
module a;	// old a

class C
{
	int fn1() {return 2;}
	int fn2() {return 3;}
}
----------------
module a;	// new a

class C
{
	int fn2() {return 3;}	// note: order reversed
	int fn1() {return 2;}
}
----------------
module main;
import a;
import std.stdio;

void main()
{
	auto c = new C;
	writef("hello world",\n);
	int i = c.fn1();
	writef("hello world ", i,\n);
}
----------------


I know next to nothing about COFF/ELF/etc. so this suggestion may be 
impossible to implement but here goes.

How about only calculate the offsets for the vtbl once, when you compile 
the class they are in. Then in the generated object file you store some 
symbols that are the offsets. When another module imports the file, the 
code is generated to make the virtual function call but the offset is 
left as an unresolved external that must be resolved at link time. 
Furthermore, this would solve many of the problems about code hiding in 
much the same way as James Dunne's suggestion



James Dunne wrote:
[...]
 
 For the closed source community, one could also invent a specially 
 formatted D import source file which declares only the non-private 
 methods of classes *and* at which offsets they can be found at in the 
 vtable.  Of course include the public fields and at which offsets they 
 are found at as well.

Dec 30 2005
parent BCS <BCS_member pathlink.com> writes:
By the way, this bug is still around.

digitalmars.D.bugs/5671
Dec 30 2005
prev sibling next sibling parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote in
[snip]
 di files' main advantage is improving build speeds for large projects.

I regularly compile 200+ files on a mediocre desktop. Takes about 3 seconds when using Build (which eats perhaps a third or a half of that time). I would not consider it worthwhile using "di" files until the file count were up in the thousands ... from that perspective, the current value of "di" appears truly nebulous. It will likely remain so for the overwhelming majority of users; for eternity. While I always applaud your progress with the language and the compiler, don't you think this stands alone as blatant feature-creep? With respect, I think we'd all be better off without -H until an actual need for it arises ~ at that point we might even have some /useful/ requirements for it, since implementation hiding (what we all requested) is apparently not what it's intended for :-) Respectfully;
Dec 29 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp1tf9$13ds$1 digitaldaemon.com...
 While I always applaud your progress with the language and the compiler, 
 don't you think this stands alone as blatant feature-creep? With respect, 
 I think we'd all be better off without -H until an actual need for it 
 arises ~ at that point we might even have some /useful/ requirements for 
 it, since implementation hiding (what we all requested) is apparently not 
 what it's intended for :-)

Since Dave had already implemented it, and it would be needed eventually anyway, there was good reason to put it in. Furthermore, most of the work involved improving the pretty-printing capability of the compiler, much of which was stubbed out or kludged and needed to be done properly anyway. The pretty-printing capability is used for things like formatting of error messages and the doc generator. (For example, now expressions get properly parenthesized when printed, and numerous problems printing numeric and string literals were fixed.) These fixes weren't necessary for DMD to compile programs successfully, but they do contribute to it looking like a polished, presentable product.
Dec 29 2005
parent John Reimer <terminal.node gmail.com> writes:
Walter Bright wrote:
 "Kris" <fu bar.com> wrote in message news:dp1tf9$13ds$1 digitaldaemon.com...
 While I always applaud your progress with the language and the compiler, 
 don't you think this stands alone as blatant feature-creep? With respect, 
 I think we'd all be better off without -H until an actual need for it 
 arises ~ at that point we might even have some /useful/ requirements for 
 it, since implementation hiding (what we all requested) is apparently not 
 what it's intended for :-)

Since Dave had already implemented it, and it would be needed eventually anyway, there was good reason to put it in. Furthermore, most of the work involved improving the pretty-printing capability of the compiler, much of which was stubbed out or kludged and needed to be done properly anyway. The pretty-printing capability is used for things like formatting of error messages and the doc generator. (For example, now expressions get properly parenthesized when printed, and numerous problems printing numeric and string literals were fixed.) These fixes weren't necessary for DMD to compile programs successfully, but they do contribute to it looking like a polished, presentable product.

That's definitely a plus. If these are the only benefits of the addition, I can appreciate them. -JJR
Dec 29 2005
prev sibling next sibling parent reply "Zz" <junkie noware.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dp1s4k$12ev$1 digitaldaemon.com...
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 In other words, let's enhance the language so we can support the needs of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we need 
 to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

There's more than that, there are template bodies. At some point, we have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

I'm not suggesting that the solution below in worth using but it's a good idea to look into. Quite some time ago I used to do some work in Modula2 and the way implementation and interface were divided was very intutive, over time I've got used to other methods and eventually felt comfortable with them. This link gives a short overview http://www.csc.twu.ca/rsbook/Ch6/Ch6.5.html Oberon2 which came later on removed this concept but it was later on re-introduced. for eg in Modula2: DEFINITION MODULE MyRealMath; CONST root2 = 1.414213562; PROCEDURE log (x : REAL) : REAL; END MyRealMath.and in another file:IMPLEMENTATION MODULE MyRealMath; (* note same name *) FROM RealMath IMPORT ln; PROCEDURE log (x : REAL) : REAL; (* returns the base 10 logarithm of x *) BEGIN RETURN (ln (x)) / (ln (10.0) ) END log; END MyRealMath.user file:MODULE Client; FROM MyRealMath IMPORT root2, log;
Dec 29 2005
parent reply John Reimer <terminal.node gmail.com> writes:
Zz wrote:
 "Walter Bright" <newshound digitalmars.com> wrote in message 
 news:dp1s4k$12ev$1 digitaldaemon.com...
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 In other words, let's enhance the language so we can support the needs of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we need 
 to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

I'm not suggesting that the solution below in worth using but it's a good idea to look into. Quite some time ago I used to do some work in Modula2 and the way implementation and interface were divided was very intutive, over time I've got used to other methods and eventually felt comfortable with them. This link gives a short overview http://www.csc.twu.ca/rsbook/Ch6/Ch6.5.html Oberon2 which came later on removed this concept but it was later on re-introduced. for eg in Modula2: DEFINITION MODULE MyRealMath; CONST root2 = 1.414213562; PROCEDURE log (x : REAL) : REAL; END MyRealMath.and in another file:IMPLEMENTATION MODULE MyRealMath; (* note same name *) FROM RealMath IMPORT ln; PROCEDURE log (x : REAL) : REAL; (* returns the base 10 logarithm of x *) BEGIN RETURN (ln (x)) / (ln (10.0) ) END log; END MyRealMath.user file:MODULE Client; FROM MyRealMath IMPORT root2, log;

I studied some Modula 2 more than 10 years ago, so I don't remember too much. But I do remember the implementation/definition file separation. Were those given different file endings? I think Modula 3 (which does somewhat the same, I believe) uses i3 for interface and m3 for implementation. It does look like an effective solution, although I don't know how such a solution would fit into a modern language like D, that has features that none of these early languages ever had. -JJR
Dec 29 2005
parent reply "Zz" <junkie noware.com> writes:
"John Reimer" <terminal.node gmail.com> wrote in message 
news:dp29ai$1a9l$1 digitaldaemon.com...
 Zz wrote:
 "Walter Bright" <newshound digitalmars.com> wrote in message 
 news:dp1s4k$12ev$1 digitaldaemon.com...
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 In other words, let's enhance the language so we can support the needs 
 of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we need 
 to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

I'm not suggesting that the solution below in worth using but it's a good idea to look into. Quite some time ago I used to do some work in Modula2 and the way implementation and interface were divided was very intutive, over time I've got used to other methods and eventually felt comfortable with them. This link gives a short overview http://www.csc.twu.ca/rsbook/Ch6/Ch6.5.html Oberon2 which came later on removed this concept but it was later on re-introduced. for eg in Modula2: DEFINITION MODULE MyRealMath; CONST root2 = 1.414213562; PROCEDURE log (x : REAL) : REAL; END MyRealMath.and in another file:IMPLEMENTATION MODULE MyRealMath; (* note same name *) FROM RealMath IMPORT ln; PROCEDURE log (x : REAL) : REAL; (* returns the base 10 logarithm of x *) BEGIN RETURN (ln (x)) / (ln (10.0) ) END log; END MyRealMath.user file:MODULE Client; FROM MyRealMath IMPORT root2, log;

I studied some Modula 2 more than 10 years ago, so I don't remember too much. But I do remember the implementation/definition file separation. Were those given different file endings? I think Modula 3 (which does somewhat the same, I believe) uses i3 for interface and m3 for implementation. It does look like an effective solution, although I don't know how such a solution would fit into a modern language like D, that has features that none of these early languages ever had.

Stony Brook and Topspeed Modula2 used .MOD for Implementation and .DEF for Definition. As for fitting within a modern language like D, I wonder why not (at least it's better than the -H option). In Stonnt Brook M2 the libs were generated from the DEF and lib and all the user needed was the lib and def to do any thing, the def module did not contain anything (just like D's interface but in a separate file with the option to include other def's). Here is an example of ctrl3d. First the DEF module: DEFINITION MODULE CTL3D; (* Notice that you can include other Modules here*) FROM WIN32 IMPORT HWND; (* functions to interface with the CTL3D32.DLL for Windows NT *) (* most function do nothing on WIN95, actually any Windows version 4.x *) PROCEDURE Ctl3dLoaded() : BOOLEAN; (* returns TRUE if CTL3D32 is loaded and initialized *) (* CTL3D32 will not be loaded if system supports DS_3DLOOK dialogs *) PROCEDURE MakeAllDialogs3d; (* if Ctl3dLoaded() = TRUE, then forces all dialogs to 3D *) END CTL3D. and now the MOD (implementation) module. Here I pasted more just to give you more detail on what is going on. IMPLEMENTATION MODULE CTL3D; <*/NOWARN:F/INLINE:N*> FROM SYSTEM IMPORT ADDRESS; FROM WIN32 IMPORT HWND, DWORD, BOOL, HINSTANCE, FARPROC, LoadLibrary, FreeLibrary, GetProcAddress, OSVERSIONINFO, GetVersionEx; FROM WINUSER IMPORT GetWindowLong, GWL_STYLE, DS_3DLOOK; FROM WINX IMPORT Instance, NULL_HINSTANCE; VAR Have3dDll : BOOLEAN; Dll : HINSTANCE; C3DRegister : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DUnregister : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DAutoSubclass : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DSubclassDlgEx : PROCEDURE(HWND, DWORD) : BOOL [WIN32SYSTEM]; C3DColorChange : PROCEDURE() : BOOL [WIN32SYSTEM]; PROCEDURE Load3dDll; BEGIN Dll := LoadLibrary("CTL3D32"); PROCEDURE MakeAllDialogs3d; BEGIN IF Have3dDll THEN C3DAutoSubclass(Instance); END; END MakeAllDialogs3d; BEGIN Have3dDll := FALSE; IF NOT Builtin3d() THEN Load3dDll; END; FINALLY IF Have3dDll THEN C3DUnregister(Instance); FreeLibrary(Dll); END; END CTL3D. Regards Zz
Dec 30 2005
next sibling parent reply rko <rko_member pathlink.com> writes:
In article <dp35ru$1vle$1 digitaldaemon.com>, Zz says...
"John Reimer" <terminal.node gmail.com> wrote in message 
news:dp29ai$1a9l$1 digitaldaemon.com...
 Zz wrote:
 "Walter Bright" <newshound digitalmars.com> wrote in message 
 news:dp1s4k$12ev$1 digitaldaemon.com...
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 In other words, let's enhance the language so we can support the needs 
 of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we need 
 to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

I'm not suggesting that the solution below in worth using but it's a good idea to look into. Quite some time ago I used to do some work in Modula2 and the way implementation and interface were divided was very intutive, over time I've got used to other methods and eventually felt comfortable with them. This link gives a short overview http://www.csc.twu.ca/rsbook/Ch6/Ch6.5.html Oberon2 which came later on removed this concept but it was later on re-introduced. for eg in Modula2: DEFINITION MODULE MyRealMath; CONST root2 = 1.414213562; PROCEDURE log (x : REAL) : REAL; END MyRealMath.and in another file:IMPLEMENTATION MODULE MyRealMath; (* note same name *) FROM RealMath IMPORT ln; PROCEDURE log (x : REAL) : REAL; (* returns the base 10 logarithm of x *) BEGIN RETURN (ln (x)) / (ln (10.0) ) END log; END MyRealMath.user file:MODULE Client; FROM MyRealMath IMPORT root2, log;

I studied some Modula 2 more than 10 years ago, so I don't remember too much. But I do remember the implementation/definition file separation. Were those given different file endings? I think Modula 3 (which does somewhat the same, I believe) uses i3 for interface and m3 for implementation. It does look like an effective solution, although I don't know how such a solution would fit into a modern language like D, that has features that none of these early languages ever had.

Stony Brook and Topspeed Modula2 used .MOD for Implementation and .DEF for Definition. As for fitting within a modern language like D, I wonder why not (at least it's better than the -H option). In Stonnt Brook M2 the libs were generated from the DEF and lib and all the user needed was the lib and def to do any thing, the def module did not contain anything (just like D's interface but in a separate file with the option to include other def's). Here is an example of ctrl3d. First the DEF module: DEFINITION MODULE CTL3D; (* Notice that you can include other Modules here*) FROM WIN32 IMPORT HWND; (* functions to interface with the CTL3D32.DLL for Windows NT *) (* most function do nothing on WIN95, actually any Windows version 4.x *) PROCEDURE Ctl3dLoaded() : BOOLEAN; (* returns TRUE if CTL3D32 is loaded and initialized *) (* CTL3D32 will not be loaded if system supports DS_3DLOOK dialogs *) PROCEDURE MakeAllDialogs3d; (* if Ctl3dLoaded() = TRUE, then forces all dialogs to 3D *) END CTL3D. and now the MOD (implementation) module. Here I pasted more just to give you more detail on what is going on. IMPLEMENTATION MODULE CTL3D; <*/NOWARN:F/INLINE:N*> FROM SYSTEM IMPORT ADDRESS; FROM WIN32 IMPORT HWND, DWORD, BOOL, HINSTANCE, FARPROC, LoadLibrary, FreeLibrary, GetProcAddress, OSVERSIONINFO, GetVersionEx; FROM WINUSER IMPORT GetWindowLong, GWL_STYLE, DS_3DLOOK; FROM WINX IMPORT Instance, NULL_HINSTANCE; VAR Have3dDll : BOOLEAN; Dll : HINSTANCE; C3DRegister : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DUnregister : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DAutoSubclass : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DSubclassDlgEx : PROCEDURE(HWND, DWORD) : BOOL [WIN32SYSTEM]; C3DColorChange : PROCEDURE() : BOOL [WIN32SYSTEM]; PROCEDURE Load3dDll; BEGIN Dll := LoadLibrary("CTL3D32"); PROCEDURE MakeAllDialogs3d; BEGIN IF Have3dDll THEN C3DAutoSubclass(Instance); END; END MakeAllDialogs3d; BEGIN Have3dDll := FALSE; IF NOT Builtin3d() THEN Load3dDll; END; FINALLY IF Have3dDll THEN C3DUnregister(Instance); FreeLibrary(Dll); END; END CTL3D. Regards Zz

i am his opinion. boy, would it be nice getting someting like the modula definition file system. rko
Dec 30 2005
parent "Zz" <junkie noware.com> writes:
 i am his opinion. boy, would it be nice getting someting like the modula
 definition file system.

 rko

It would be a good feature to have but I don't believe it will ever be available in D. Whenever someone hears of a language that Wirth designed they assume that we are talking about Pascal and don't look any furher. Stonny Brook M2 used this system to do their cross platform stuff and in use it worked perfectly, and I don't ever recall having a problem with it. On the other hand D is a new language which and I really do enjoy doing small quick and dirty things with it (I would never dream of doing a large project with it witout some form of separation, maybe it's just me but I can't see it happening). Note: I have some stuff done in D running in a commercial production enviroment, but they were small programs. Regards, Zz
Dec 30 2005
prev sibling next sibling parent rko <rko_member pathlink.com> writes:
In article <dp35ru$1vle$1 digitaldaemon.com>, Zz says...
"John Reimer" <terminal.node gmail.com> wrote in message 
news:dp29ai$1a9l$1 digitaldaemon.com...
 Zz wrote:
 "Walter Bright" <newshound digitalmars.com> wrote in message 
 news:dp1s4k$12ev$1 digitaldaemon.com...
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 In other words, let's enhance the language so we can support the needs 
 of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we need 
 to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

I'm not suggesting that the solution below in worth using but it's a good idea to look into. Quite some time ago I used to do some work in Modula2 and the way implementation and interface were divided was very intutive, over time I've got used to other methods and eventually felt comfortable with them. This link gives a short overview http://www.csc.twu.ca/rsbook/Ch6/Ch6.5.html Oberon2 which came later on removed this concept but it was later on re-introduced. for eg in Modula2: DEFINITION MODULE MyRealMath; CONST root2 = 1.414213562; PROCEDURE log (x : REAL) : REAL; END MyRealMath.and in another file:IMPLEMENTATION MODULE MyRealMath; (* note same name *) FROM RealMath IMPORT ln; PROCEDURE log (x : REAL) : REAL; (* returns the base 10 logarithm of x *) BEGIN RETURN (ln (x)) / (ln (10.0) ) END log; END MyRealMath.user file:MODULE Client; FROM MyRealMath IMPORT root2, log;

I studied some Modula 2 more than 10 years ago, so I don't remember too much. But I do remember the implementation/definition file separation. Were those given different file endings? I think Modula 3 (which does somewhat the same, I believe) uses i3 for interface and m3 for implementation. It does look like an effective solution, although I don't know how such a solution would fit into a modern language like D, that has features that none of these early languages ever had.

Stony Brook and Topspeed Modula2 used .MOD for Implementation and .DEF for Definition. As for fitting within a modern language like D, I wonder why not (at least it's better than the -H option). In Stonnt Brook M2 the libs were generated from the DEF and lib and all the user needed was the lib and def to do any thing, the def module did not contain anything (just like D's interface but in a separate file with the option to include other def's). Here is an example of ctrl3d. First the DEF module: DEFINITION MODULE CTL3D; (* Notice that you can include other Modules here*) FROM WIN32 IMPORT HWND; (* functions to interface with the CTL3D32.DLL for Windows NT *) (* most function do nothing on WIN95, actually any Windows version 4.x *) PROCEDURE Ctl3dLoaded() : BOOLEAN; (* returns TRUE if CTL3D32 is loaded and initialized *) (* CTL3D32 will not be loaded if system supports DS_3DLOOK dialogs *) PROCEDURE MakeAllDialogs3d; (* if Ctl3dLoaded() = TRUE, then forces all dialogs to 3D *) END CTL3D. and now the MOD (implementation) module. Here I pasted more just to give you more detail on what is going on. IMPLEMENTATION MODULE CTL3D; <*/NOWARN:F/INLINE:N*> FROM SYSTEM IMPORT ADDRESS; FROM WIN32 IMPORT HWND, DWORD, BOOL, HINSTANCE, FARPROC, LoadLibrary, FreeLibrary, GetProcAddress, OSVERSIONINFO, GetVersionEx; FROM WINUSER IMPORT GetWindowLong, GWL_STYLE, DS_3DLOOK; FROM WINX IMPORT Instance, NULL_HINSTANCE; VAR Have3dDll : BOOLEAN; Dll : HINSTANCE; C3DRegister : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DUnregister : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DAutoSubclass : PROCEDURE(HINSTANCE) : BOOL [WIN32SYSTEM]; C3DSubclassDlgEx : PROCEDURE(HWND, DWORD) : BOOL [WIN32SYSTEM]; C3DColorChange : PROCEDURE() : BOOL [WIN32SYSTEM]; PROCEDURE Load3dDll; BEGIN Dll := LoadLibrary("CTL3D32"); PROCEDURE MakeAllDialogs3d; BEGIN IF Have3dDll THEN C3DAutoSubclass(Instance); END; END MakeAllDialogs3d; BEGIN Have3dDll := FALSE; IF NOT Builtin3d() THEN Load3dDll; END; FINALLY IF Have3dDll THEN C3DUnregister(Instance); FreeLibrary(Dll); END; END CTL3D. Regards Zz

here is a def file for a class (i took out some of the stuff): DEFINITION MODULE database; FROM SQL IMPORT *; IMPORT EXCEPTIONS; TYPE SQL_ENV; CONST XALLOCERROR = -2001; CLASS DATABASE; REVEAL StrBaseToCard64, .... , Commit; TYPE NumberType = (Annex, Filter); VAR myConnection: SQL_ENV; dbUsed : DBTYPE; ExSource : EXCEPTIONS.ExceptionSource; PROCEDURE MakeGenericNewNumber(tableName: ARRAY OF CHAR) : CARDINAL64; .... PROCEDURE Commit() : INTEGER32; END DATABASE; END database.
Dec 30 2005
prev sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Zz" <junkie noware.com> wrote in message 
news:dp35ru$1vle$1 digitaldaemon.com...
 In Stonnt Brook M2 the libs were generated from the DEF and lib and all 
 the user needed was the lib and def to do any thing, the def module did 
 not contain anything (just like D's interface but in a separate file with 
 the option to include other def's).

The trouble with that is the usual trouble with interfaces - they necessarilly introduce an extra level of indirection to get at the methods/data. If there's no way around it, the language implementation is never going to attain the needed performance.
Jan 03 2006
parent ZZ <ZZ zz.com> writes:
Walter Bright wrote:
 "Zz" <junkie noware.com> wrote in message 
 news:dp35ru$1vle$1 digitaldaemon.com...
 
In Stonnt Brook M2 the libs were generated from the DEF and lib and all 
the user needed was the lib and def to do any thing, the def module did 
not contain anything (just like D's interface but in a separate file with 
the option to include other def's).

The trouble with that is the usual trouble with interfaces - they necessarilly introduce an extra level of indirection to get at the methods/data. If there's no way around it, the language implementation is never going to attain the needed performance.

You have a point about the extra level of indirection, SB M2 allowed one to put procedures into the DEF (interface file) and they would be inlined in case someone needed that. Just like C++ inline functions. I guess it all depends on background, I introduced one of the people I work with to D (he has to re-write something small I wrote in D to C++) and at first glance the thing that he loved was that it didn't have header files (I wrote the whole thing in one file). There is a new thread "Decoupling implementation from usage" that helped clear some things up for me. All in all I've come to really like D and will continue using it. Thanks Zz
Jan 03 2006
prev sibling parent reply John Reimer <terminal.node gmail.com> writes:
Walter Bright wrote:
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...
 In other words, let's enhance the language so we can support the needs of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we need to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

There's more than that, there are template bodies. At some point, we have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

Okay, now I see why there is so much confusion going on with the addition of this feature. There are two problems to be solved: one is the issue of implementation hiding; the other is the issue of compiler performance. I think most of us mistook the di file generator to be a solution for implementation hiding for closed source projects. Obviously we got the wrong idea about the whole thing. If di files serve the same purpose as "precompiled headers," obviously there is no consideration whatsoever for implementation hiding (and there never was any such intention). Good to know. Personally, I was less interested in compile performance and more interested in implementation hiding. The -H flag, therefore, will not be of much use to me. -JJR
Dec 29 2005
next sibling parent reply J C Calvarese <technocrat7 gmail.com> writes:
In article <dp27tj$19f5$1 digitaldaemon.com>, John Reimer says...
Okay, now I see why there is so much confusion going on with the 
addition of this feature.  There are two problems to be solved: one is 
the issue of implementation hiding; the other is the issue of compiler 
performance.  I think most of us mistook the di file generator to be a 
solution for implementation hiding for closed source projects. 
Obviously we got the wrong idea about the whole thing.

If di files serve the same purpose as "precompiled headers," obviously 
there is no consideration whatsoever for implementation hiding (and 
there never was any such intention).

Good to know.  Personally, I was less interested in compile performance 
and more interested in implementation hiding.  The -H flag, therefore, 
will not be of much use to me.

-JJR

I have to say as fast as DMD compiles files, I was more interested in seeing headers stripped to the bone than optimally stripped, too (I really don't want that to sound as weird as it does, but I think "stripped aggressively" would be worse). I doubt that I would personally use it for hiding code (my code isn't interesting enough to bother hiding), but it seems like a neat thing to do. But I was thinking... If the magic happens in the (open source) front-end, it might not be that hard for an enterprising individual to alter the compiler to leave out the optimizations. I found in "mars.c" where the command line option "-H" is detected, and I followed the program flow to "hdrgen.c". But that's as far as I got before I lost interest. Time for sleep anyway. ;) jcc7
Dec 29 2005
parent John Reimer <terminal.node gmail.com> writes:
J C Calvarese wrote:
 In article <dp27tj$19f5$1 digitaldaemon.com>, John Reimer says...
 Okay, now I see why there is so much confusion going on with the 
 addition of this feature.  There are two problems to be solved: one is 
 the issue of implementation hiding; the other is the issue of compiler 
 performance.  I think most of us mistook the di file generator to be a 
 solution for implementation hiding for closed source projects. 
 Obviously we got the wrong idea about the whole thing.

 If di files serve the same purpose as "precompiled headers," obviously 
 there is no consideration whatsoever for implementation hiding (and 
 there never was any such intention).

 Good to know.  Personally, I was less interested in compile performance 
 and more interested in implementation hiding.  The -H flag, therefore, 
 will not be of much use to me.

 -JJR

I have to say as fast as DMD compiles files, I was more interested in seeing headers stripped to the bone than optimally stripped, too (I really don't want that to sound as weird as it does, but I think "stripped aggressively" would be worse). I doubt that I would personally use it for hiding code (my code isn't interesting enough to bother hiding), but it seems like a neat thing to do. But I was thinking... If the magic happens in the (open source) front-end, it might not be that hard for an enterprising individual to alter the compiler to leave out the optimizations. I found in "mars.c" where the command line option "-H" is detected, and I followed the program flow to "hdrgen.c". But that's as far as I got before I lost interest. Time for sleep anyway. ;) jcc7

It's probably possible. Or they could just start on a project separate from the compiler that creates the headers. Like Carlos mentioned, digc may already do this. -JJR
Dec 30 2005
prev sibling next sibling parent reply Kyle Furlong <kylefurlong gmail.com> writes:
John Reimer wrote:
 Walter Bright wrote:
 
 "Derek Parnell" <derek psych.ward> wrote in message 
 news:zqfcc85x3d7z$.1n4xv1fmu7njt$.dlg 40tude.net...

 In other words, let's enhance the language so we can support the 
 needs of
 developers who want closed-source development *and* the needs of the
 compiler to generate correctly performing code. This would mean we 
 need to
 resolve the issue of class member offsets, inlining, and const
 declarations. These are not impossible to resolve.

There's more than that, there are template bodies. At some point, we have to ask if we've just got the wrong approach to implementation hiding. I suggest that interface classes are the way to get true implementation hiding, not di files. di files' main advantage is improving build speeds for large projects. They work a lot like "precompiled headers" do in C++.

Okay, now I see why there is so much confusion going on with the addition of this feature. There are two problems to be solved: one is the issue of implementation hiding; the other is the issue of compiler performance. I think most of us mistook the di file generator to be a solution for implementation hiding for closed source projects. Obviously we got the wrong idea about the whole thing. If di files serve the same purpose as "precompiled headers," obviously there is no consideration whatsoever for implementation hiding (and there never was any such intention). Good to know. Personally, I was less interested in compile performance and more interested in implementation hiding. The -H flag, therefore, will not be of much use to me. -JJR

Right on John. Walter, what is your answer for closed source implementation hiding?
Dec 30 2005
parent reply John Reimer <terminal.node gmail.com> writes:
Kyle Furlong wrote:
 John Reimer wrote:

 Okay, now I see why there is so much confusion going on with the 
 addition of this feature.  There are two problems to be solved: one is 
 the issue of implementation hiding; the other is the issue of compiler 
 performance.  I think most of us mistook the di file generator to be a 
 solution for implementation hiding for closed source projects. 
 Obviously we got the wrong idea about the whole thing.

 If di files serve the same purpose as "precompiled headers," obviously 
 there is no consideration whatsoever for implementation hiding (and 
 there never was any such intention).

 Good to know.  Personally, I was less interested in compile 
 performance and more interested in implementation hiding.  The -H 
 flag, therefore, will not be of much use to me.

 -JJR

Right on John. Walter, what is your answer for closed source implementation hiding?

I think Walter already answered this question in a previous post in this thread. If I understand correctly, he basically detailed how implementation hiding is nigh to impossible to do effectively. I'm not sure if he meant that it's not worth trying to program a solution because data can't be effectively hidden or that the difficultly is not worth the effort. I just want to clarify that my idea of implementation-hiding really didn't entail encrypted libraries. I know somebody could find out implementation details if they really wanted to. The idea, much like the Modula 2 and 3 interface definition files, is merely to separate the details: (1) so that it isn't /easy/ to see implementation details and (2) so that we have a summary of the features a library provides. A nifty side effect is that there's probably increase compiler performance. In the end though, maybe something like DDL (www.dsource.org) will solve all our problems in some way. Eliminating the need for the compiler to parse a header altogether may be a big step forward. Also, while I feel that the -H option seems to be more of distraction from fixing higher priority items in D, it may do well for us at some point; I'm just hoping, though, that all the bugs it introduces don't take up Walter's time. A tool like this could theoretically be separate from the compiler. That may be the best solution in the end if it's too much work to maintain. I vaguely remember arguments to this end being made months to a year or so before. -JJR
Dec 30 2005
parent reply "Kris" <fu bar.com> writes:
Just to clarify a couple of minor things:

As John noted, implementation hiding is not about encryption ~ it's about 
supporting closed-source development.

D /does/ support implementation hiding, and has done for a long time. What's 
been missing is a nice, simple, automated, standard, and compiler-supported 
way to generate a bridge between the "hidden" implementation, and client 
usage thereof. It's always been done manually, and it's easy to get confused 
over which file is which since there's common module names involved etc.

One way to approach information hiding is to use Interfaces in conjunction 
with a Factory pattern. There's been a couple of trivial examples posted in 
this thread, but the general idea is that Interfaces are used to abstract 
away all implementation details, and one or more public functions are 
exposed in a "bridging module" to create instances on a clients behalf.

Another approach to implementation hiding is to simply strip out existing 
method bodies and leave the bare declarations, along with any enums and/or 
typedefs they may use. Many requests have been made for an automated tool to 
do exactly this: which would also work well for the Interface approach noted 
above. The end result would be the same in both cases ~ a secondary 
"declaration only" file would be produced ~ which would be used by a client 
as a means to get the appropriate code linked. In all cases, the client is 
expected to know nothing of, nor care about, the implementation under the 
hood.

This is where a misunderstanding arose with -H which, as it turns out, has 
no real value with respect to the above. Thus, D does not have a tool to 
help produce a client-binding from an implementation ~ you /still/ have to 
do that manually. That doesn't mean it can't be done, but it can certainly 
be a chore, and it would be good to get some standardization in the 
mechanics of it (such as a standard file-extension, for example, such that 
the conflict over module names can be better managed).

So ~

1) D still needs a much more solid *story* for closed-source development. 
Without it, one could argue it deserves to remain a niche language. Note 
that this is not a technical hurdle, but it's one that has repeatedly failed 
to be passed. All it needs is a little TLC and attention.

2) The -H flag is apparently for something else entirely.


Hope that helps?

- Kris



"John Reimer" <terminal.node gmail.com> wrote
 Kyle Furlong wrote:
 John Reimer wrote:

 Okay, now I see why there is so much confusion going on with the 
 addition of this feature.  There are two problems to be solved: one is 
 the issue of implementation hiding; the other is the issue of compiler 
 performance.  I think most of us mistook the di file generator to be a 
 solution for implementation hiding for closed source projects. Obviously 
 we got the wrong idea about the whole thing.

 If di files serve the same purpose as "precompiled headers," obviously 
 there is no consideration whatsoever for implementation hiding (and 
 there never was any such intention).

 Good to know.  Personally, I was less interested in compile performance 
 and more interested in implementation hiding.  The -H flag, therefore, 
 will not be of much use to me.

 -JJR

Right on John. Walter, what is your answer for closed source implementation hiding?

I think Walter already answered this question in a previous post in this thread. If I understand correctly, he basically detailed how implementation hiding is nigh to impossible to do effectively. I'm not sure if he meant that it's not worth trying to program a solution because data can't be effectively hidden or that the difficultly is not worth the effort. I just want to clarify that my idea of implementation-hiding really didn't entail encrypted libraries. I know somebody could find out implementation details if they really wanted to. The idea, much like the Modula 2 and 3 interface definition files, is merely to separate the details: (1) so that it isn't /easy/ to see implementation details and (2) so that we have a summary of the features a library provides. A nifty side effect is that there's probably increase compiler performance. In the end though, maybe something like DDL (www.dsource.org) will solve all our problems in some way. Eliminating the need for the compiler to parse a header altogether may be a big step forward. Also, while I feel that the -H option seems to be more of distraction from fixing higher priority items in D, it may do well for us at some point; I'm just hoping, though, that all the bugs it introduces don't take up Walter's time. A tool like this could theoretically be separate from the compiler. That may be the best solution in the end if it's too much work to maintain. I vaguely remember arguments to this end being made months to a year or so before. -JJR

Dec 30 2005
next sibling parent "Kris" <fu bar.com> writes:
I forgot to note that DDL does not help here at all. On the contrary, DDL 
firmly places a generously sized spotlight directly upon the issue.

- Kris


"Kris" <fu bar.com> wrote in message news:dp3sve$2hm8$1 digitaldaemon.com...
 Just to clarify a couple of minor things:

 As John noted, implementation hiding is not about encryption ~ it's about 
 supporting closed-source development.

 D /does/ support implementation hiding, and has done for a long time. 
 What's been missing is a nice, simple, automated, standard, and 
 compiler-supported way to generate a bridge between the "hidden" 
 implementation, and client usage thereof. It's always been done manually, 
 and it's easy to get confused over which file is which since there's 
 common module names involved etc.

 One way to approach information hiding is to use Interfaces in conjunction 
 with a Factory pattern. There's been a couple of trivial examples posted 
 in this thread, but the general idea is that Interfaces are used to 
 abstract away all implementation details, and one or more public functions 
 are exposed in a "bridging module" to create instances on a clients 
 behalf.

 Another approach to implementation hiding is to simply strip out existing 
 method bodies and leave the bare declarations, along with any enums and/or 
 typedefs they may use. Many requests have been made for an automated tool 
 to do exactly this: which would also work well for the Interface approach 
 noted above. The end result would be the same in both cases ~ a secondary 
 "declaration only" file would be produced ~ which would be used by a 
 client as a means to get the appropriate code linked. In all cases, the 
 client is expected to know nothing of, nor care about, the implementation 
 under the hood.

 This is where a misunderstanding arose with -H which, as it turns out, has 
 no real value with respect to the above. Thus, D does not have a tool to 
 help produce a client-binding from an implementation ~ you /still/ have to 
 do that manually. That doesn't mean it can't be done, but it can certainly 
 be a chore, and it would be good to get some standardization in the 
 mechanics of it (such as a standard file-extension, for example, such that 
 the conflict over module names can be better managed).

 So ~

 1) D still needs a much more solid *story* for closed-source development. 
 Without it, one could argue it deserves to remain a niche language. Note 
 that this is not a technical hurdle, but it's one that has repeatedly 
 failed to be passed. All it needs is a little TLC and attention.

 2) The -H flag is apparently for something else entirely.


 Hope that helps?

 - Kris



 "John Reimer" <terminal.node gmail.com> wrote
 Kyle Furlong wrote:
 John Reimer wrote:

 Okay, now I see why there is so much confusion going on with the 
 addition of this feature.  There are two problems to be solved: one is 
 the issue of implementation hiding; the other is the issue of compiler 
 performance.  I think most of us mistook the di file generator to be a 
 solution for implementation hiding for closed source projects. 
 Obviously we got the wrong idea about the whole thing.

 If di files serve the same purpose as "precompiled headers," obviously 
 there is no consideration whatsoever for implementation hiding (and 
 there never was any such intention).

 Good to know.  Personally, I was less interested in compile performance 
 and more interested in implementation hiding.  The -H flag, therefore, 
 will not be of much use to me.

 -JJR

Right on John. Walter, what is your answer for closed source implementation hiding?

I think Walter already answered this question in a previous post in this thread. If I understand correctly, he basically detailed how implementation hiding is nigh to impossible to do effectively. I'm not sure if he meant that it's not worth trying to program a solution because data can't be effectively hidden or that the difficultly is not worth the effort. I just want to clarify that my idea of implementation-hiding really didn't entail encrypted libraries. I know somebody could find out implementation details if they really wanted to. The idea, much like the Modula 2 and 3 interface definition files, is merely to separate the details: (1) so that it isn't /easy/ to see implementation details and (2) so that we have a summary of the features a library provides. A nifty side effect is that there's probably increase compiler performance. In the end though, maybe something like DDL (www.dsource.org) will solve all our problems in some way. Eliminating the need for the compiler to parse a header altogether may be a big step forward. Also, while I feel that the -H option seems to be more of distraction from fixing higher priority items in D, it may do well for us at some point; I'm just hoping, though, that all the bugs it introduces don't take up Walter's time. A tool like this could theoretically be separate from the compiler. That may be the best solution in the end if it's too much work to maintain. I vaguely remember arguments to this end being made months to a year or so before. -JJR


Dec 30 2005
prev sibling next sibling parent John Reimer <terminal.node gmail.com> writes:
Kris wrote:
 Just to clarify a couple of minor things:
 
 As John noted, implementation hiding is not about encryption ~ it's about 
 supporting closed-source development.
 
 D /does/ support implementation hiding, and has done for a long time. What's 
 been missing is a nice, simple, automated, standard, and compiler-supported 
 way to generate a bridge between the "hidden" implementation, and client 
 usage thereof. It's always been done manually, and it's easy to get confused 
 over which file is which since there's common module names involved etc.
 
 One way to approach information hiding is to use Interfaces in conjunction 
 with a Factory pattern. There's been a couple of trivial examples posted in 
 this thread, but the general idea is that Interfaces are used to abstract 
 away all implementation details, and one or more public functions are 
 exposed in a "bridging module" to create instances on a clients behalf.
 
 Another approach to implementation hiding is to simply strip out existing 
 method bodies and leave the bare declarations, along with any enums and/or 
 typedefs they may use. Many requests have been made for an automated tool to 
 do exactly this: which would also work well for the Interface approach noted 
 above. The end result would be the same in both cases ~ a secondary 
 "declaration only" file would be produced ~ which would be used by a client 
 as a means to get the appropriate code linked. In all cases, the client is 
 expected to know nothing of, nor care about, the implementation under the 
 hood.
 
 This is where a misunderstanding arose with -H which, as it turns out, has 
 no real value with respect to the above. Thus, D does not have a tool to 
 help produce a client-binding from an implementation ~ you /still/ have to 
 do that manually. That doesn't mean it can't be done, but it can certainly 
 be a chore, and it would be good to get some standardization in the 
 mechanics of it (such as a standard file-extension, for example, such that 
 the conflict over module names can be better managed).
 
 So ~
 
 1) D still needs a much more solid *story* for closed-source development. 
 Without it, one could argue it deserves to remain a niche language. Note 
 that this is not a technical hurdle, but it's one that has repeatedly failed 
 to be passed. All it needs is a little TLC and attention.
 
 2) The -H flag is apparently for something else entirely.
 
 
 Hope that helps?
 
 - Kris
 
 

Well said, Kris. Much better than I could put into words. Thanks! -JJR
Dec 30 2005
prev sibling next sibling parent reply "Dave" <Dave_member pathlink.com> writes:
"Kris" <fu bar.com> wrote in message news:dp3sve$2hm8$1 digitaldaemon.com...
 Just to clarify a couple of minor things:

 As John noted, implementation hiding is not about encryption ~ it's about
 supporting closed-source development.

 D /does/ support implementation hiding, and has done for a long time.
 What's been missing is a nice, simple, automated, standard, and
 compiler-supported way to generate a bridge between the "hidden"
 implementation, and client usage thereof. It's always been done manually,
 and it's easy to get confused over which file is which since there's
 common module names involved etc.

 One way to approach information hiding is to use Interfaces in conjunction
 with a Factory pattern. There's been a couple of trivial examples posted
 in this thread, but the general idea is that Interfaces are used to
 abstract away all implementation details, and one or more public functions
 are exposed in a "bridging module" to create instances on a clients
 behalf.

 Another approach to implementation hiding is to simply strip out existing
 method bodies and leave the bare declarations, along with any enums and/or
 typedefs they may use. Many requests have been made for an automated tool
 to do exactly this: which would also work well for the Interface approach
 noted above. The end result would be the same in both cases ~ a secondary

This is basically what -H does, except that inlinable function bodies are emitted. As pointed out earlier, private members often need to be there too, with or without function bodies (offsets, const initializers, templates), otherwise it gets pretty complicated, fast. I experimented with most/all the implementation hiding stuff and got most of it working, I think, but in the end after testing it would have taken a whole heck of a lot more work (and time maintaining it) to make and keep the thing bulletproof (see: digitalmars.D/29883). Why don't you give -H a try by using it against phobos, DWT and Mango (once the integration issues are taken care of)? Then see if it still really makes the implementation details all that apparent (or at least more apparent than C/++ library header files do now)? - Dave
 "declaration only" file would be produced ~ which would be used by a
 client as a means to get the appropriate code linked. In all cases, the
 client is expected to know nothing of, nor care about, the implementation
 under the hood.

 This is where a misunderstanding arose with -H which, as it turns out, has
 no real value with respect to the above. Thus, D does not have a tool to
 help produce a client-binding from an implementation ~ you /still/ have to
 do that manually. That doesn't mean it can't be done, but it can certainly
 be a chore, and it would be good to get some standardization in the
 mechanics of it (such as a standard file-extension, for example, such that
 the conflict over module names can be better managed).

 So ~

 1) D still needs a much more solid *story* for closed-source development.
 Without it, one could argue it deserves to remain a niche language. Note
 that this is not a technical hurdle, but it's one that has repeatedly
 failed to be passed. All it needs is a little TLC and attention.

 2) The -H flag is apparently for something else entirely.


 Hope that helps?

 - Kris



 "John Reimer" <terminal.node gmail.com> wrote
 Kyle Furlong wrote:
 John Reimer wrote:

 Okay, now I see why there is so much confusion going on with the
 addition of this feature.  There are two problems to be solved: one is
 the issue of implementation hiding; the other is the issue of compiler
 performance.  I think most of us mistook the di file generator to be a
 solution for implementation hiding for closed source projects.
 Obviously we got the wrong idea about the whole thing.

 If di files serve the same purpose as "precompiled headers," obviously
 there is no consideration whatsoever for implementation hiding (and
 there never was any such intention).

 Good to know.  Personally, I was less interested in compile performance
 and more interested in implementation hiding.  The -H flag, therefore,
 will not be of much use to me.

 -JJR

Right on John. Walter, what is your answer for closed source implementation hiding?

I think Walter already answered this question in a previous post in this thread. If I understand correctly, he basically detailed how implementation hiding is nigh to impossible to do effectively. I'm not sure if he meant that it's not worth trying to program a solution because data can't be effectively hidden or that the difficultly is not worth the effort. I just want to clarify that my idea of implementation-hiding really didn't entail encrypted libraries. I know somebody could find out implementation details if they really wanted to. The idea, much like the Modula 2 and 3 interface definition files, is merely to separate the details: (1) so that it isn't /easy/ to see implementation details and (2) so that we have a summary of the features a library provides. A nifty side effect is that there's probably increase compiler performance. In the end though, maybe something like DDL (www.dsource.org) will solve all our problems in some way. Eliminating the need for the compiler to parse a header altogether may be a big step forward. Also, while I feel that the -H option seems to be more of distraction from fixing higher priority items in D, it may do well for us at some point; I'm just hoping, though, that all the bugs it introduces don't take up Walter's time. A tool like this could theoretically be separate from the compiler. That may be the best solution in the end if it's too much work to maintain. I vaguely remember arguments to this end being made months to a year or so before. -JJR


Dec 30 2005
parent reply "Kris" <fu bar.com> writes:
Perhaps there's further misunderstanding here:

"Dave" <Dave_member pathlink.com> wrote in
 "Kris" <fu bar.com> wrote ...
 Another approach to implementation hiding is to simply strip out existing
 method bodies and leave the bare declarations, along with any enums 
 and/or
 typedefs they may use. Many requests have been made for an automated tool
 to do exactly this: which would also work well for the Interface approach
 noted above. The end result would be the same in both cases ~ a secondary

This is basically what -H does, except that inlinable function bodies are emitted. As pointed out earlier, private members often need to be there too, with or without function bodies (offsets, const initializers, templates), otherwise it gets pretty complicated, fast.

And because of that, it really has no value for implementation hiding. I'd go as far as to say it simply complicates things instead. Do imports get exposed also? I'd try this myself, but the compiler GPFs. Further, exposing public attributes in this manner seems to defy logic, and appears to breaks every visibility rule in the book. Such declarations were made private because they were intended to be, /ahem/, private?
 I experimented with most/all the implementation hiding stuff and got most 
 of it working, I think, but in the end after testing it would have taken a 
 whole heck of a lot more work (and time maintaining it) to make and keep 
 the thing bulletproof (see: 
 digitalmars.D/29883).

 Why don't you give -H a try by using it against phobos, DWT and Mango 
 (once the integration issues are taken care of)? Then see if it still 
 really makes the implementation details all that apparent (or at least 
 more apparent than C/++ library header files do now)?

Forgive me Dave, but I just can't see how that's relevant. What you describe is not what, for example, any Fortune-500 company would consider usable. Walter apparently agrees. In reference to software contract-exposure: ".di files aren't necessarilly going to help you here." Then there's the question of what this feature thinks it addresses. Walter seems to feel it's about compiler performance: "di files' main advantage is improving build speeds for large projects", and this is borne out in the flurry of posts. You also intimated this in a recent post. Yet you're also saying it can be used for implementation-hiding? Pardon me, but I have to disagree on all points: <rant> 1) -H has little or no benefit for implementation hiding. It may look as though it could suffice, but I'll be happy to bet the current incarnation is a questionable toy in this regard. If I could actually get it to work, then I'd be happy to prove the point. Perhaps surprisingly, Walter appears to agree. 2) -H certainly ought to cut down on import size. So, just how much does that affect the compiler performance? It's already blazingly fast. I'll be a monkey's uncle if -H makes true /real-world/ differences to more than 5% of users. On the other hand, a /binary/ symbolic representation (as discussed several times) could make a really notable dent for certain environments, if such speedups were to become a necessity. If -H is all about performance, then why isn't it based around such a representation? 3) You speculatively mentioned DWT and RAD IDEs. There's a few things to consider there: (a) if the API is stable, the headers can be produced by a standalone tool ~ it doesn't need to be part of the compiler; (b) it the headers in question are being revised, they need to be regenerated on the fly; and (c) a RAD tool would likely prefer to have a library made available that hides all the details of creating and managing such headers, and would likely prefer that happened at the binary-symbolic level, explicitly for reasons of performance. Especially in case (a). Partial source-code munging does little to help such a tool ~ it's just not going far enough. I've no intent on causing discomfort to you or anyone here, but I really have to say this feature appears to have been given not /nearly/ sufficient thought. It fails to support industrial-level implementation-hiding (the one place it is needed) and it doesn't push the performance envelope very hard at all. You can call me an ass if you like, Dave, but this is surely feature-creep of the worst kind ~ there's apparently no meat. Can you, or anyone, please help me grasp where the real value lies? It's not that I'm somehow against new functionality. Part of what bugs me about this is the reference compiler sets an unwritten "standard" that others must somehow comply with. Whatever those standards are, they surely ought to be well considered. </rant> I can point to one side-effect that actually does help: the compiler now looks for "di" files, which means that hand-coded "implementation bridging" files can (at last!) live side-by-side with the real implementation module, eliminating some prior grief for a developer. However, this conflicts with -H rather badly, and therefore is likely to be an unintended and somewhat precarious benefit. Was that aspect considered? Respectfully; - Kris
Dec 30 2005
next sibling parent John Reimer <terminal.node gmail.com> writes:
Kris wrote:
 Perhaps there's further misunderstanding here:

 I can point to one side-effect that actually does help: the compiler now 
 looks for "di" files, which means that hand-coded "implementation bridging" 
 files can (at last!) live side-by-side with the real implementation module, 
 eliminating some prior grief for a developer. However, this conflicts 
 with -H rather badly, and therefore is likely to be an unintended and 
 somewhat precarious benefit. Was that aspect considered?
 

Interesting! I didn't really think of that aspect. That could be useful, indeed. Though, as you said, it probably is precarious. -JJR
Dec 30 2005
prev sibling next sibling parent reply Sean Kelly <sean f4.ca> writes:
Kris wrote:
 
 And because of that, it really has no value for implementation hiding. I'd 
 go as far as to say it simply complicates things instead. Do imports get 
 exposed also? I'd try this myself, but the compiler GPFs. Further, exposing 
 public attributes in this manner seems to defy logic, and appears to breaks 
 every visibility rule in the book. Such declarations were made private 
 because they were intended to be, /ahem/, private?

But, as Walter stated, private data might affect alignment and such so it must be exposed. If implementation hiding is an issue and if interfaces are not appropriate, the pimpl idiom from C++ might be worth considering.
 Forgive me Dave, but I just can't see how that's relevant. What you describe 
 is not what, for example, any Fortune-500 company would consider usable. 
 Walter apparently agrees. In reference to software contract-exposure: ".di 
 files aren't necessarilly going to help you here."
 
 Then there's the question of what this feature thinks it addresses. Walter 
 seems to feel it's about compiler performance: "di files' main advantage is 
 improving build speeds for large projects", and this is borne out in the 
 flurry of posts. You also intimated this in a recent post. Yet you're also 
 saying it can be used for implementation-hiding?

I think the most relevant comparison is with C/C++ header files, as D headers are nearly identical in what they provide. And as Walter said, the primary use of headers is to speed compilation. That implementation can be concealed in some cases is secondary. Another concern is template files. C++ has 'export' in an attempt to address this, but it's a tremendously complex feature that provides questionable benefit. I think Walter is right for not attempting anything like this for D quite yet.
 Pardon me, but I have to disagree on all points:
 
 <rant>
 
 1) -H has little or no benefit for implementation hiding. It may look as 
 though it could suffice, but I'll be happy to bet the current incarnation is 
 a questionable toy in this regard. If I could actually get it to work, then 
 I'd be happy to prove the point. Perhaps surprisingly, Walter appears to 
 agree.

See my comment about the pimpl (pointer to implementation) idea above. This is commonly used both to obscure implementation details and to reduce coupling.
 
 2) -H certainly ought to cut down on import size. So, just how much does 
 that affect the compiler performance? It's already blazingly fast. I'll be a 
 monkey's uncle if -H makes true /real-world/ differences to more than 5% of 
 users. On the other hand, a /binary/ symbolic representation (as discussed 
 several times) could make a really notable dent for certain environments, if 
 such speedups were to become a necessity. If -H is all about performance, 
 then why isn't it based around such a representation?

Another issue is that header files are touched far less frequently than implementation files. Thus, implementation changes can be made and so long as the header remains unchanged the compiler doesn't have to rebuild dependent modules. In large projects, this can be of tremendous value. To this end, it might be nice if DMD only touched header files for nonzero deltas (so timestamps only changed if the file contents changed).
 3) You speculatively mentioned DWT and RAD IDEs. There's a few things to 
 consider there: (a) if the API is stable, the headers can be produced by a 
 standalone tool ~ it doesn't need to be part of the compiler; (b) it the 
 headers in question are being revised, they need to be regenerated on the 
 fly; and (c) a RAD tool would likely prefer to have a library made available 
 that hides all the details of creating and managing such headers, and would 
 likely prefer that happened at the binary-symbolic level, explicitly for 
 reasons of performance. Especially in case (a). Partial source-code munging 
 does little to help such a tool ~ it's just not going far enough.

True enough. But since a front-end is really required to do the job, why not build the feature into the compiler? Particularly since each compiler may have different metrics for inlining and such.
 I've no intent on causing discomfort to you or anyone here, but I really 
 have to say this feature appears to have been given not /nearly/ sufficient 
 thought. It fails to support industrial-level implementation-hiding (the one 
 place it is needed) and it doesn't push the performance envelope very hard 
 at all.

I see implementation hiding as a separate (though related) QOI issue. Typically, I want to see readable header files and let things get mangled in post-production before the CD is sent out to clients.
 You can call me an ass if you like, Dave, but this is surely feature-creep 
 of the worst kind ~ there's apparently no meat. Can you, or anyone, please 
 help me grasp where the real value lies? It's not that I'm somehow against 
 new functionality. Part of what bugs me about this is the reference compiler 
 sets an unwritten "standard" that others must somehow comply with. Whatever 
 those standards are, they surely ought to be well considered.

I think it's important to separate QOI features from "standard" features. Will it be necessary for all D compilers to have code coverage or profiling tools? Of course not. Header generation is in the same category.
 I can point to one side-effect that actually does help: the compiler now 
 looks for "di" files, which means that hand-coded "implementation bridging" 
 files can (at last!) live side-by-side with the real implementation module, 
 eliminating some prior grief for a developer. However, this conflicts 
 with -H rather badly, and therefore is likely to be an unintended and 
 somewhat precarious benefit. Was that aspect considered?

I haven't installed this new version of DMD yet so I don't know much about how -H works yet. What I said above is based on what I've read here today :-) Sean
Dec 30 2005
next sibling parent reply "Kris" <fu bar.com> writes:
First, I must point out that this branch is about "implementation hiding" ~ 
it say so in the topic :)
Second, if there's a point in this reply, it's at the end.


"Sean Kelly" <sean f4.ca> wrote
 But, as Walter stated, private data might affect alignment and such so it 
 must be exposed.  If implementation hiding is an issue and if interfaces 
 are not appropriate, the pimpl idiom from C++ might be worth considering.

Interfaces are in the same boat, Sean. D still needs something nice and easy to generate the "implementation bridge" module that folk have been asking after for a long time. You can always build it by hand, but there's always been problems of one kind or another. Thankfully, a side-effect of -H will help. I'm not familiar with pimpl.
 I think the most relevant comparison is with C/C++ header files, as D 
 headers are nearly identical in what they provide.  And as Walter said, 
 the primary use of headers is to speed compilation.  That implementation 
 can be concealed in some cases is secondary.  Another concern is template 
 files.  C++ has 'export' in an attempt to address this, but it's a 
 tremendously complex feature that provides questionable benefit. I think 
 Walter is right for not attempting anything like this for D quite yet.

Then let's call the darned thing a "performance enhancement" and stop all this continued pretense :)
 Another issue is that header files are touched far less frequently than 
 implementation files.  Thus, implementation changes can be made and so 
 long as the header remains unchanged the compiler doesn't have to rebuild 
 dependent modules.  In large projects, this can be of tremendous value. 
 To this end, it might be nice if DMD only touched header files for nonzero 
 deltas (so timestamps only changed if the file contents changed).

In C, yes. D currently does not have any such issues.
 I see implementation hiding as a separate (though related) QOI issue. 
 Typically, I want to see readable header files and let things get mangled 
 in post-production before the CD is sent out to clients.

Ack ~ this is not about mangled headers!
 You can call me an ass if you like, Dave, but this is surely 
 feature-creep of the worst kind ~ there's apparently no meat. Can you, or 
 anyone, please help me grasp where the real value lies? It's not that I'm 
 somehow against new functionality. Part of what bugs me about this is the 
 reference compiler sets an unwritten "standard" that others must somehow 
 comply with. Whatever those standards are, they surely ought to be well 
 considered.

I think it's important to separate QOI features from "standard" features. Will it be necessary for all D compilers to have code coverage or profiling tools? Of course not. Header generation is in the same category.

Is it? If, just if, people start using -H in an attempt at implementation-hiding, then other compilers will have to follow suit. After all, code will be written to take advantage of those headers (the whole point of implementation-hiding), and thus will be dependent on the way the information is exposed. Thus, once we start down that road others will have to comply.
Dec 30 2005
parent reply Sean Kelly <sean f4.ca> writes:
Kris wrote:
 First, I must point out that this branch is about "implementation hiding" ~ 
 it say so in the topic :)

Sorry :-) I suppose I read too much into some of the text I omitted.
 Another issue is that header files are touched far less frequently than 
 implementation files.  Thus, implementation changes can be made and so 
 long as the header remains unchanged the compiler doesn't have to rebuild 
 dependent modules.  In large projects, this can be of tremendous value. 
 To this end, it might be nice if DMD only touched header files for nonzero 
 deltas (so timestamps only changed if the file contents changed).

In C, yes. D currently does not have any such issues.

How so? I'll admit I haven't looked into how the need for recompilation is detected.
 I see implementation hiding as a separate (though related) QOI issue. 
 Typically, I want to see readable header files and let things get mangled 
 in post-production before the CD is sent out to clients.

Ack ~ this is not about mangled headers!

I had a few things going through my head and unfortunately only one of them made it out :-) I was thinking of implementation hiding in .NET where mangling is pretty much necessary. So what would be fitting then? The ability to generate pure headers with no inlined function calls and no private data?
 I think it's important to separate QOI features from "standard" features. 
 Will it be necessary for all D compilers to have code coverage or 
 profiling tools?  Of course not.  Header generation is in the same 
 category.

Is it? If, just if, people start using -H in an attempt at implementation-hiding, then other compilers will have to follow suit. After all, code will be written to take advantage of those headers (the whole point of implementation-hiding), and thus will be dependent on the way the information is exposed. Thus, once we start down that road others will have to comply.

I'm not sure I understand. The -H option must necessarily generate headers that contain much of the same symbolic information as the original module or the header will not be usable. Hrm... perhaps instead of containing private data for alignment purposes, a pragma could be used to indicate data size? ie. class C { private: int x; int y; } // translates to class C { pragma( pad, 8 ); }
Dec 30 2005
parent "Kris" <fu bar.com> writes:
"Sean Kelly" <sean f4.ca> wrote...
 Sorry :-)  I suppose I read too much into some of the text I omitted.

Mea cupla :)
 Another issue is that header files are touched far less frequently than 
 implementation files.  Thus, implementation changes can be made and so 
 long as the header remains unchanged the compiler doesn't have to 
 rebuild dependent modules.  In large projects, this can be of tremendous 
 value. To this end, it might be nice if DMD only touched header files 
 for nonzero deltas (so timestamps only changed if the file contents 
 changed).

In C, yes. D currently does not have any such issues.

How so? I'll admit I haven't looked into how the need for recompilation is detected.

I'm sorry. I meant that D doesn't have those performance issues. Introducing D header files does, of course, introduce the synchronization problem you note. But this performance aspect is a different one from implementation hiding.
 Ack ~ this is not about mangled headers!

I had a few things going through my head and unfortunately only one of them made it out :-) I was thinking of implementation hiding in .NET where mangling is pretty much necessary. So what would be fitting then? The ability to generate pure headers with no inlined function calls and no private data?

That's effectively what so many requests have been about. Imported modules must be considered also. In a closed-source, proprietary environments, one shouldn't be exposing various bits of implementation or private attributes; certainly when there's no legitimate reason to do so.
 I think it's important to separate QOI features from "standard" 
 features. Will it be necessary for all D compilers to have code coverage 
 or profiling tools?  Of course not.  Header generation is in the same 
 category.

Is it? If, just if, people start using -H in an attempt at implementation-hiding, then other compilers will have to follow suit. After all, code will be written to take advantage of those headers (the whole point of implementation-hiding), and thus will be dependent on the way the information is exposed. Thus, once we start down that road others will have to comply.

I'm not sure I understand. The -H option must necessarily generate headers that contain much of the same symbolic information as the original module or the header will not be usable. Hrm... perhaps instead of containing private data for alignment purposes, a pragma could be used to indicate data size? ie. class C { private: int x; int y; } // translates to class C { pragma( pad, 8 ); }

Pardon me if I copy a bit from another post: "I have to say, though, that's there's a fundamental flaw exposed by trying to please two masters: you can't do implementation-hiding correctly whilst exposing partial-implementation and private attributes. And you can't provide for faster-compilation without doing so." That's a generalization, and refers to inline-code concerns, templates, and so on. To go one further: If D cannot, technically, expose a module or a class for external usage without exposing various proprietary private attributes and proprietary snippets of code (which in turn may reference proprietary yet private imports), then there's something very far wrong with the entire concept of D. This would cause DDL a blow, and put into question the legitimacy of using D for commercial purpose. I have, and continue to assume this is not the case. We should refer back to Derek's example, but I'll paste it here instead (hope Derek doesn't mind): ========================= This source file ... //-------------------------------- import std.stdio; private int qwerty; class A { private int x; void ths() { x = 1; } } void xpub() { A a = new A; qwerty = 1; } private void xpriv() { A b = new A; qwerty = 2; } //-------------------------------- was turned into this 'header' ... // D import file generated from 'test.d' import std.stdio; private { int qwerty; } class A { private { int x; } void ths() { x = 1; } } void xpub() { A a = new A; qwerty = 1; } private { void xpriv() { A b = new A; qwerty = 2; } } //------------------------- but I was expecting something more like this ... // D import file generated by hand import std.stdio; class A { void ths(){} } void xpub(){} =================== The latter is what we've been asking for (for a long time). Note that Derek also included a (non-private) import in the hand-built version? Private imports should not be exposed.
Dec 30 2005
prev sibling parent "Walter Bright" <newshound digitalmars.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message 
news:dp4r60$8e5$1 digitaldaemon.com...
 I think it's important to separate QOI features from "standard" features. 
 Will it be necessary for all D compilers to have code coverage or 
 profiling tools?  Of course not.  Header generation is in the same 
 category.

That's right. I tried to make that distiction explicit in the doc. -H is not a D language feature, it's a DMD implementation feature (and one GDC will likely pick up as well).
Dec 30 2005
prev sibling next sibling parent reply JT <JT_member pathlink.com> writes:
In article <dp4oq5$6qt$1 digitaldaemon.com>, Kris says...
Perhaps there's further misunderstanding here:

"Dave" <Dave_member pathlink.com> wrote in
 "Kris" <fu bar.com> wrote ...
 Another approach to implementation hiding is to simply strip out existing
 method bodies and leave the bare declarations, along with any enums 
 and/or
 typedefs they may use. Many requests have been made for an automated tool
 to do exactly this: which would also work well for the Interface approach
 noted above. The end result would be the same in both cases ~ a secondary

This is basically what -H does, except that inlinable function bodies are emitted. As pointed out earlier, private members often need to be there too, with or without function bodies (offsets, const initializers, templates), otherwise it gets pretty complicated, fast.

And because of that, it really has no value for implementation hiding. I'd go as far as to say it simply complicates things instead. Do imports get exposed also? I'd try this myself, but the compiler GPFs. Further, exposing public attributes in this manner seems to defy logic, and appears to breaks every visibility rule in the book. Such declarations were made private because they were intended to be, /ahem/, private?
 I experimented with most/all the implementation hiding stuff and got most 
 of it working, I think, but in the end after testing it would have taken a 
 whole heck of a lot more work (and time maintaining it) to make and keep 
 the thing bulletproof (see: 
 digitalmars.D/29883).

 Why don't you give -H a try by using it against phobos, DWT and Mango 
 (once the integration issues are taken care of)? Then see if it still 
 really makes the implementation details all that apparent (or at least 
 more apparent than C/++ library header files do now)?

Forgive me Dave, but I just can't see how that's relevant. What you describe is not what, for example, any Fortune-500 company would consider usable. Walter apparently agrees. In reference to software contract-exposure: ".di files aren't necessarilly going to help you here." Then there's the question of what this feature thinks it addresses. Walter seems to feel it's about compiler performance: "di files' main advantage is improving build speeds for large projects", and this is borne out in the flurry of posts. You also intimated this in a recent post. Yet you're also saying it can be used for implementation-hiding? Pardon me, but I have to disagree on all points: <rant> 1) -H has little or no benefit for implementation hiding. It may look as though it could suffice, but I'll be happy to bet the current incarnation is a questionable toy in this regard. If I could actually get it to work, then I'd be happy to prove the point. Perhaps surprisingly, Walter appears to agree. 2) -H certainly ought to cut down on import size. So, just how much does that affect the compiler performance? It's already blazingly fast. I'll be a monkey's uncle if -H makes true /real-world/ differences to more than 5% of users. On the other hand, a /binary/ symbolic representation (as discussed several times) could make a really notable dent for certain environments, if such speedups were to become a necessity. If -H is all about performance, then why isn't it based around such a representation? 3) You speculatively mentioned DWT and RAD IDEs. There's a few things to consider there: (a) if the API is stable, the headers can be produced by a standalone tool ~ it doesn't need to be part of the compiler; (b) it the headers in question are being revised, they need to be regenerated on the fly; and (c) a RAD tool would likely prefer to have a library made available that hides all the details of creating and managing such headers, and would likely prefer that happened at the binary-symbolic level, explicitly for reasons of performance. Especially in case (a). Partial source-code munging does little to help such a tool ~ it's just not going far enough. I've no intent on causing discomfort to you or anyone here, but I really have to say this feature appears to have been given not /nearly/ sufficient thought. It fails to support industrial-level implementation-hiding (the one place it is needed) and it doesn't push the performance envelope very hard at all. You can call me an ass if you like, Dave, but this is surely feature-creep of the worst kind ~ there's apparently no meat. Can you, or anyone, please help me grasp where the real value lies? It's not that I'm somehow against new functionality. Part of what bugs me about this is the reference compiler sets an unwritten "standard" that others must somehow comply with. Whatever those standards are, they surely ought to be well considered. </rant> I can point to one side-effect that actually does help: the compiler now looks for "di" files, which means that hand-coded "implementation bridging" files can (at last!) live side-by-side with the real implementation module, eliminating some prior grief for a developer. However, this conflicts with -H rather badly, and therefore is likely to be an unintended and somewhat precarious benefit. Was that aspect considered? Respectfully; - Kris

I think part of the confusion is lack of knowledge of how the DMD compiler works. When one compiles a DMD module - it parses and performs full analysis on all imported modules - and all modules they import etc. Theres a section of comments refering to importing only pre-compiled symbols for imported objects instead of just parsing the complete source. I believe this is some sort if middle ground between importing symbols and parsing the source. And I can very much see how it could be usefull in huge projects but I personally havent experienced compile times that are not lighting fast. Also lets not forget what walter mentioned about this feature fleshing out the code generation output of the compiler which will be usefull down the road. I just dont understand why people react with such negativity and disrespect towards walter if he implements a feature that they dont need. Its really immature. Some of you act like you are paying Walter or something. What ever.
Dec 30 2005
parent reply "Kris" <fu bar.com> writes:
"JT" <JT_member pathlink.com> wrote
 I think part of the confusion is lack of knowledge of how the DMD compiler
 works. When one compiles a DMD module - it parses and performs full 
 analysis on
 all imported modules - and all modules they import etc.

I think most folks understand this. Is it an issue?
 Theres a section of
 comments refering to importing only pre-compiled symbols for imported 
 objects
 instead of just parsing the complete source. I believe this is some sort 
 if
 middle ground between importing symbols and parsing the source.

The "middle ground" you speak of is exactly that. It's not needed at this point, so why burden the state of play with adding such a thing? It's called feature creep.
 And I can very
 much see how it could be usefull in huge projects but I personally havent
 experienced compile times that are not lighting fast.

There's no question that a binary-symbolic implementation would be great. But, as you point out, there's no pressing need for it right now. Each time I compile, it's with 200+ files on a mediocre machine. I just timed it, and it takes just over 2 seconds. That includes the time Build takes. The compiler is very, very, fast. I think that's indicitive of a good design and a lot of valuable experience. The same cannot be said for -H, since it doesn't even seem to be clear about what it addresses. Also lets not forget what
 walter mentioned about this feature fleshing out the code generation 
 output of
 the compiler which will be usefull down the road.

Sure. But that's not what's being addressed. What it being addressed is the latest claim that this is somehow for implementation-hiding. It is simply not. Performance wise? Different subject matter.
 I just dont understand why
 people react with such negativity and disrespect towards walter if he 
 implements
 a feature that they dont need. Its really immature. Some of you act like 
 you are
 paying Walter or something. What ever.

First, I don't see where there was /any/ negativity or disrespect towards Walter in that post. It was adressed entirely towards Dave. But then, so what if it were ardently disagreeing with Walter directly? Walter is a tough nut to crack ... there's no value in mincing words with him. You are, of course, welcome to your opinion. As am I. My opinion is this: The community has a rather large stake in the future of D. This is why Walter has this NG ~ it's a way to create the evangelizers, and get the message out. D is a grass-root effort. There are a few handfuls of people who have contributed an /extraordinary/ amount of their /own/ time, in many different ways. Whether it is through ideas and suggestions, through encouraging other to try D, through hosting the NG, through hosting dsource.org, writing libraries or building tools. Do you think the community has no claim whatsoever to the D language? Do you feel we should not be able to shape its direction? Or criticize it? Chastise it? Thus, we're not paying Walter to write the compiler. Nor is he paying those who contribute so much. It goes both ways, and it's pretty immature of you to state otherwise. I'm sure you'd agree.
Dec 30 2005
parent reply Sean Kelly <sean f4.ca> writes:
Kris wrote:
 "JT" <JT_member pathlink.com> wrote
 I think part of the confusion is lack of knowledge of how the DMD compiler
 works. When one compiles a DMD module - it parses and performs full 
 analysis on
 all imported modules - and all modules they import etc.

I think most folks understand this. Is it an issue?

Ah, and this is what you meant about timestamping not being used in D. Works for me. Sean
Dec 30 2005
parent reply Sean Kelly <sean f4.ca> writes:
Sean Kelly wrote:
 Kris wrote:
 "JT" <JT_member pathlink.com> wrote
 I think part of the confusion is lack of knowledge of how the DMD 
 compiler
 works. When one compiles a DMD module - it parses and performs full 
 analysis on
 all imported modules - and all modules they import etc.

I think most folks understand this. Is it an issue?

Ah, and this is what you meant about timestamping not being used in D. Works for me.

I posted without thinking about this :-p. While this may be how the compiler works (and approximately how any compiler works, so far as I'm aware), this is separate from the timestamping issue. Sean
Dec 30 2005
parent reply JT <JT_member pathlink.com> writes:
In article <dp53dk$e44$1 digitaldaemon.com>, Sean Kelly says...
Sean Kelly wrote:
 Kris wrote:
 "JT" <JT_member pathlink.com> wrote
 I think part of the confusion is lack of knowledge of how the DMD 
 compiler
 works. When one compiles a DMD module - it parses and performs full 
 analysis on
 all imported modules - and all modules they import etc.

I think most folks understand this. Is it an issue?

Ah, and this is what you meant about timestamping not being used in D. Works for me.

I posted without thinking about this :-p. While this may be how the compiler works (and approximately how any compiler works, so far as I'm aware), this is separate from the timestamping issue. Sean

well in C++ you only parse headers.... thats all this -H is doing. its providing a way to compile large projects without having to parse every module for every module etc. I still cant understand why people are upset because a feature exists they dont need. Its downright comedy.
Dec 30 2005
next sibling parent reply "Kris" <fu bar.com> writes:
"JT" <JT_member pathlink.com> wrote
  I still cant understand why people are upset because a feature
 exists they dont need. Its downright comedy.

It would be, if it had no ramifications. It's clear that you're thinking about the claimed "performance enhancing" aspects alone, and not the real topic at all. If you really think that I personally fit into your little frame above, then you're sorely mistaken. Sure is nice that you're being entertained, though :)
Dec 30 2005
parent reply JT <JT_member pathlink.com> writes:
In article <dp574e$gie$1 digitaldaemon.com>, Kris says...
"JT" <JT_member pathlink.com> wrote
  I still cant understand why people are upset because a feature
 exists they dont need. Its downright comedy.

It would be, if it had no ramifications. It's clear that you're thinking about the claimed "performance enhancing" aspects alone, and not the real topic at all. If you really think that I personally fit into your little frame above, then you're sorely mistaken. Sure is nice that you're being entertained, though :)

Well I agree with a lot of what you are saying but I think you clearly want a *DIFFERENT* solution for implementation hiding than -H. Its like people are saying "Walter, this apple tastes like an orange!". maybe thats because it is an orange. :D Maybe you need to use the front end and come up with your own solution? just a suggestion....
Dec 30 2005
parent "Kris" <fu bar.com> writes:
Indeed. We should try to be clear that -H performs some kind of source-code 
munging for purposes other than this topic. Several folk (including myself) 
have tried to say, in a variety of different ways, "we should desist from 
associating it with implementation-hiding, since it is not currently 
targeted for that task".

As you say, it is currently "an orange" :)




"JT" <JT_member pathlink.com> wrote in message 
news:dp57m1$gs6$1 digitaldaemon.com...
 In article <dp574e$gie$1 digitaldaemon.com>, Kris says...
"JT" <JT_member pathlink.com> wrote
  I still cant understand why people are upset because a feature
 exists they dont need. Its downright comedy.

It would be, if it had no ramifications. It's clear that you're thinking about the claimed "performance enhancing" aspects alone, and not the real topic at all. If you really think that I personally fit into your little frame above, then you're sorely mistaken. Sure is nice that you're being entertained, though :)

Well I agree with a lot of what you are saying but I think you clearly want a *DIFFERENT* solution for implementation hiding than -H. Its like people are saying "Walter, this apple tastes like an orange!". maybe thats because it is an orange. :D Maybe you need to use the front end and come up with your own solution? just a suggestion....

Dec 30 2005
prev sibling parent "Derek Parnell" <derek psych.ward> writes:
On Sat, 31 Dec 2005 16:44:18 +1100, JT <JT_member pathlink.com> wrote:

 I still cant understand why people are upset because a feature
 exists they dont need. Its downright comedy.

It's because of a misunderstanding. Myself and some others assumed its purpose was to support closed-source development, especially by the way it was presented by Walter, but it turns out that it's purpose is to speed up compilation when libraries are being used. I'm afraid its really just because we misunderstood Walter's (and Dave's) intent for the feature. Personally, I have no need for it as compilations speeds are so fast that even a 50% reduction would have zero effect on productivity. -- Derek Parnell Melbourne, Australia
Dec 31 2005
prev sibling next sibling parent reply Dave <Dave_member pathlink.com> writes:
In article <dp4oq5$6qt$1 digitaldaemon.com>, Kris says...
Perhaps there's further misunderstanding here:

"Dave" <Dave_member pathlink.com> wrote in
 "Kris" <fu bar.com> wrote ...
 Another approach to implementation hiding is to simply strip out existing
 method bodies and leave the bare declarations, along with any enums 
 and/or
 typedefs they may use. Many requests have been made for an automated tool
 to do exactly this: which would also work well for the Interface approach
 noted above. The end result would be the same in both cases ~ a secondary


I've no intent on causing discomfort to you or anyone here, but I really have to say this feature appears to have been given not /nearly/ sufficient thought. It fails to support industrial-level implementation-hiding (the one place it is needed) and it doesn't push the performance envelope very hard at all.

No discomfort at all - Everyone has their opinions. This obviously doesn't have to be the be-all and end-all to the 'header', symbol file, what-have-you functionality, and I don't think it was ever intended as such, certainly not by me. At the very least it brought out the discussion -- included in my earlier post was a link to an older request for functionality and no-one replied, so I implemented what seemed to make the most sense and what I had time for, at least as a *first step*. I would applaud anyone (Kris?, Derek?) who would take what's there and provide a better solution -- plain-and-simple fact is I've spent all the time I can on it. - Dave
Dec 30 2005
next sibling parent "Kris" <fu bar.com> writes:
"Dave" <Dave_member pathlink.com> wrote
 No discomfort at all - Everyone has their opinions.

I'm very glad you feel that way.
 This obviously doesn't have
 to be the be-all and end-all to the 'header', symbol file, what-have-you
 functionality, and I don't think it was ever intended as such, certainly 
 not by
 me.

Hey; it's tough to please anyone these days :)
 At the very least it brought out the discussion -- included in my earlier 
 post
 was a link to an older request for functionality and no-one replied, so I
 implemented what seemed to make the most sense and what I had time for, at 
 least
 as a *first step*.

 I would applaud anyone (Kris?, Derek?) who would take what's there and 
 provide a
 better solution -- plain-and-simple fact is I've spent all the time I can 
 on it.

It's certainly refreshing to hear such down-to-earth statements. Unfortunately, it's proven nigh impossible to get fixes into the darned library (phobos). It'll be harder still to get fixes/changes into the front-end. Perhaps you'll share your secret :) I have to say, though, that's there's a fundamental flaw exposed by trying to please two masters: you can't do implementation-hiding correctly whilst exposing partial-implementation and private attributes. And you can't provide for faster-compilation without doing so. While I personally admire you tremendously for taking the initiative (no mean feat), I have to take serious issue with the imp-hiding aspect, since it has long-term ramifications. Regards;
Dec 30 2005
prev sibling parent reply J C Calvarese <technocrat7 gmail.com> writes:
In article <dp50iu$bna$1 digitaldaemon.com>, Dave says...
In article <dp4oq5$6qt$1 digitaldaemon.com>, Kris says...
Perhaps there's further misunderstanding here:

"Dave" <Dave_member pathlink.com> wrote in



..
I've no intent on causing discomfort to you or anyone here, but I really 
have to say this feature appears to have been given not /nearly/ sufficient 
thought. It fails to support industrial-level implementation-hiding (the one 
place it is needed) and it doesn't push the performance envelope very hard 
at all.

No discomfort at all - Everyone has their opinions. This obviously doesn't have to be the be-all and end-all to the 'header', symbol file, what-have-you functionality, and I don't think it was ever intended as such, certainly not by me. At the very least it brought out the discussion -- included in my earlier post was a link to an older request for functionality and no-one replied, so I implemented what seemed to make the most sense and what I had time for, at >least as a *first step*. I would applaud anyone (Kris?, Derek?) who would take what's there and provide >a better solution -- plain-and-simple fact is I've spent all the time I can on >it. - Dave

Dave, thanks for taking all of the negative criticism so well. (I'm sure I wouldn't be able to maintain my calm in the face of so many surprisingly severe posts.) I think the main issue might be that one or more people are jealous that you not only implemented something that you find useful, but also convinced Walter to add it to the compiler. I once posted a bug report (including the simple fix) that went unacknowleged for months. ;) I don't see the real harm in the "-H" switch. Since I guess .di files are read first, I can see that people will need to make sure that their .di files stay in synch with their .d files, but that could be solved with a batch file that negates any possible speed-up benefits are the "-H" switch. It's the same issue that happens when I install lib's into \dmd\lib and need to remember to copy new libs to reap the benefits from the changes I've tried. On the other hand, this new switch will give us a new way to analyze D code with. We could see what is a candidate for inlining and what isn't. That seems like it could be pretty useful to me. I just hope that if people feel so strongly about the need to have implementation-hiding in D that they'll spend a little more time trying to hack up the front-end to reach their goals (and a little less time ranting about it in the newsgroup). jcc7
Jan 01 2006
parent reply Dave <Dave_member pathlink.com> writes:
In article <dp9bci$jcn$1 digitaldaemon.com>, J C Calvarese says...
I think the main issue might be that one or more people are jealous that you not
only implemented something that you find useful, but also convinced Walter to
add it to the compiler. I once posted a bug report (including the simple fix)
that went unacknowleged for months. ;)

I don't think I convinced Walter of anything here <g> More of "coinciding opinion" probably. I think he saw a need for something offered as a contribution already implemented and tested, so he could spend the time on other more important stuff. Hey, it was enjoyable and instructive for me <g>
On the other hand, this new switch will give us a new way to analyze D code
with. We could see what is a candidate for inlining and what isn't. That seems
like it could be pretty useful to me.

Hopefully it will eventually be seen as more useful than just inline feedback (although that part is kind of cool too) <g> As mentioned elsewhere, it cuts the code for the compiler to process by 80% for a library like DWT. I agree with Walter that for large projects and such the value of this should become more apparent as D is used in those environments more. Other people have mentioned binary representation, symbol files, etc... IMHO, the problem there is that you'll need a tool to read (and debug) those and/or the 'symbol' representation will be implemented differently between compilers. Unless, of course, you first specify an "intermediate language" which really wouldn't hide anything anyway. It's taken 5+ years to specify D already - how long for the IL ;) At least this is legal and readable D code, is automatically generated, and really should not give away any more implementation details than comparable C/++ header files do now.
I just hope that if people feel so strongly about the need to have
implementation-hiding in D that they'll spend a little more time trying to hack
up the front-end to reach their goals (and a little less time ranting about it
in the newsgroup).

I'm not saying this to discourage anyone from the effort, but AFAIK most/all of the things mentioned in this thread, have been tried and abandoned for various reasons (see: digitalmars.D/29883). It was a circuitous route. <g> It'd be nice if someone out there can come up with something better since obviously it is very important to some people and probably somewhat important to many others.
jcc7

Jan 01 2006
next sibling parent J C Calvarese <technocrat7 gmail.com> writes:
In article <dpa8f8$195s$1 digitaldaemon.com>, Dave says...

.snip...

Hopefully it will eventually be seen as more useful than just inline feedback
(although that part is kind of cool too) <g>

I wasn't try to put it down. I get that it's cool, but I think most of its benefits are somewhat over my head. ;) ..snip...
I just hope that if people feel so strongly about the need to have
implementation-hiding in D that they'll spend a little more time trying to
>>hack up the front-end to reach their goals (and a little less time ranting
>>about it in the newsgroup).

I'm not saying this to discourage anyone from the effort, but AFAIK most/all of the things mentioned in this thread, have been tried and abandoned for various reasons (see: digitalmars.D/29883). It was a circuitous route. <g>

I've read that post a few times now, but I don't know if it's sunk in yet. It's not that don't believe you. The thing is I don't really need implementation hiding for my own purposes, so I'm not trying to get in the middle of this argument. From reading the posts on this topics, I think that more implementation can be hidden. It's more a question of "Do the people that want it want it enough to get it done?" No, not everything should or could be hidden. Templates would have to be visible. If someone doesn't care about inlining, couldn't he drop a few function bodies that would be inlineable otherwise? And hiding the implementation wouldn't prevent reverse engineering (though reverse engineering might be more difficult). Or maybe I'm wrong about this, and anything else is infeasible. Sometimes, I just have to convince myself that something is impossible before I can admit what's obvious to someone else.
It'd be nice if someone out there can come up with something better since
obviously it is very important to some people and probably somewhat important
>to many others.

Sure. jcc7
Jan 01 2006
prev sibling parent "Kris" <fu bar.com> writes:
Dave ~ a couple of comments:

"Dave" <Dave_member pathlink.com> wrote
 Other people have mentioned binary representation, symbol files, etc... 
 IMHO,
 the problem there is that you'll need a tool to read (and debug) those 
 and/or
 the 'symbol' representation will be implemented differently between 
 compilers.
 Unless, of course, you first specify an "intermediate language" which 
 really
 wouldn't hide anything anyway. It's taken 5+ years to specify D already - 
 how
 long for the IL ;)

I fully agree with your assertion on the time-scale of things. While I feel that this aspect of the compiler is premature-optimization, that's just my opinion ... not that it makes a blind bit of difference anyway. As noted before, and fwiw, I truly commend you for taking the initiative and actually /doing something/! Just for the record, I recall it was Walter who said he would be doing a binary-symbolic representation.
 At least this is legal and readable D code, is automatically generated, 
 and
 really should not give away any more implementation details than 
 comparable C/++
 header files do now.

Walter has now noted that he feels it's "impractical" to mechanically perform the kind of implementation-hiding we'd been asking for, and thus we have to still do that manually. That's cool, but sometimes it's like pulling-teeth just trying to get some clarity around here. I'd like to refer back to the original posting on this subject-line (branch), since I think the perspective is relevant there? This branch is not about the "optimization" aspect, although it apparantly got mixed in.
 I'm not saying this to discourage anyone from the effort, but AFAIK 
 most/all of
 the things mentioned in this thread, have been tried and abandoned for 
 various
 reasons (see: 
 digitalmars.D/29883).
 It was a circuitous route. <g>

I wish I'd seen that post on Nov 12th ~ would have immediately noted (regarding the last paragraph of the main thrust) that emitting source code for 'inlineable' would mean potentially exposing private import modules. That's not a "good thing"tm for the kind of implementation-hiding we'd been asking for (for a long time). It also seems from your post that you had more in mind than what's currently exposed.
 obviously it is very important to some people and probably somewhat 
 important to
 many others.

It can be acutely important from a commercial standpoint. Because most people are simply tinkering with the language, you won't see much in the way of opinion there. Nor much from the open-source community. We should all understand that there's a lot of inertia against change in IT. To get a foot into the commercial segment, D needs as much help as it can possibly get in order to avoid being marginalized ~ even just keeping the darned lawyers out of the way. Implementation-hiding is one such aspect that helps. I can tell you that D is /not/ being considered where I work. Hence, I have a beef with certain weak attributes of the language and tools. I hope this clarifies certain aspects further; and thank you for being open-minded on the subject.
Jan 02 2006
prev sibling next sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp4oq5$6qt$1 digitaldaemon.com...
 Forgive me Dave, but I just can't see how that's relevant. What you 
 describe is not what, for example, any Fortune-500 company would consider 
 usable.

I disagree. Take a look at C++ header files: full class 'private' members, inlined function bodies, full template bodies, etc., all must be there. It's the same issue (which is what led to the demand for exported templates, which failed). Furthermore, Java is well accepted despite no practical hiding at all. COM/interface/factory methodology in C++ does enable full hiding, however, and if you do that in D, you get full hiding as well. Essentially, to get *practical* implementation hiding, you have to use some form of generic indirection. Successful implementation hiding schemes have all used this in one form or another: hooking interrupts in DOS, handles in the early Mac API, double indirection needed to access DLL methods, and COM programming.
Dec 30 2005
next sibling parent "Kris" <fu bar.com> writes:
Absolutely agreed; but -H does not do this for the "indirection" case 
either. Nobody is asking for miracles, but the fact remains that Derek's 
example shows that -H is hopelessly lame at hiding the implementation. If I 
had a compiler that worked, I could show you exactly what I mean.

-H is no good for implementation hiding. We should stop trying to describe 
it as such when you yourself have said as much in prior posts.

Respectfully;

- Kris



"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dp52vk$dqu$1 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dp4oq5$6qt$1 digitaldaemon.com...
 Forgive me Dave, but I just can't see how that's relevant. What you 
 describe is not what, for example, any Fortune-500 company would consider 
 usable.

I disagree. Take a look at C++ header files: full class 'private' members, inlined function bodies, full template bodies, etc., all must be there. It's the same issue (which is what led to the demand for exported templates, which failed). Furthermore, Java is well accepted despite no practical hiding at all. COM/interface/factory methodology in C++ does enable full hiding, however, and if you do that in D, you get full hiding as well. Essentially, to get *practical* implementation hiding, you have to use some form of generic indirection. Successful implementation hiding schemes have all used this in one form or another: hooking interrupts in DOS, handles in the early Mac API, double indirection needed to access DLL methods, and COM programming.

Dec 30 2005
prev sibling parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote
 COM/interface/factory methodology in C++ does enable full hiding, however, 
 and if you do that in D, you get full hiding as well.

With respect to indirection: let's go back to the Factory example I posted earlier, which the compiler failed on: ~~~~~~~~~~~~~~~~~~~ SearchInterfaces.d ~~~~~~~~~~~~~~~~~~~ module SearchInterfaces; interface ISearchEngine { ISearchItem[] lookup (char[] terms); } interface ISearchItem { int relevancy(); char[] retrievalUrl(); char[] displaySnippet(); } ~~~~~~~~~~~~~~~~~~~ SearchFactory.d ~~~~~~~~~~~~~~~~~~~ module SearchFactory; public import SearchInterfaces; // private import a.bunch.of.proprietary.stuff; // a factory for hooking up a client to the hidden implementation ISearchEngine createSearchEngine (char[] someAttributes) { // potentially reference a.bunch.of.proprietary.stuff; return new XYX(someAttributes). } // do some proprietary setup private static this() { // potentially do a.bunch.of.proprietary.stuff; } ~~~~~~~~~~~~~~~~~~~~~ That's the Factory implementation, which has a bunch of proprietary stuff in it along with references to proprietary imports. To correctly decouple a client from this implementation, one should provide another file (I'm covering Walter's statements here, from a long time ago) along the following lines: ~~~~~~~~~~~~~~~~~~~ SearchFactory.di ~~~~~~~~~~~~~~~~~~~ module SearchFactory; // I believe this has to be the same name? public import SearchInterfaces; ISearchEngine createSearchEngine (char[] someAttributes); ~~~~~~~~~~~~~~~~~~~ This setup does the right thing from an implementation-hiding aspect. It's what's been asked for many times in the past. Note that only public decalarations have been exposed. There's no class in there at all, just to make it as simple as possible. Unfortunately, I can't show the -H version. However, it's not much of a stretch to map the above onto Derek's original example, and assert that (for example) the implementation for both the factory method, and the static ctor() would be exposed (along with any other method considered to be inline-able) That would be no good, Walter. If it were the case, we'd have to ship all those proprietary imports too, along with everything they referenced ~ ad infinitum. I do wish I had a working compiler so this could be more concrete.
Dec 30 2005
next sibling parent Sean Kelly <sean f4.ca> writes:
Kris wrote:
 
 This setup does the right thing from an implementation-hiding aspect. It's 
 what's been asked for many times in the past. Note that only public 
 decalarations have been exposed. There's no class in there at all, just to 
 make it as simple as possible. Unfortunately, I can't show the -H version. 
 However, it's not much of a stretch to map the above onto Derek's original 
 example, and assert that (for example) the implementation for both the 
 factory method, and the static ctor() would be exposed (along with any other 
 method considered to be inline-able) That would be no good, Walter. If it 
 were the case, we'd have to ship all those proprietary imports too, along 
 with everything they referenced ~ ad infinitum.

So why not provide a method to turn off inlined function generation? Since inlining isn't always desired, this seems a reasonable request. Sean
Dec 30 2005
prev sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp56kj$gc8$1 digitaldaemon.com...
 This setup does the right thing from an implementation-hiding aspect. It's 
 what's been asked for many times in the past. Note that only public 
 decalarations have been exposed. There's no class in there at all, just to 
 make it as simple as possible. Unfortunately, I can't show the -H version. 
 However, it's not much of a stretch to map the above onto Derek's original 
 example, and assert that (for example) the implementation for both the 
 factory method, and the static ctor() would be exposed (along with any 
 other method considered to be inline-able) That would be no good, Walter. 
 If it were the case, we'd have to ship all those proprietary imports too, 
 along with everything they referenced ~ ad infinitum.

 I do wish I had a working compiler so this could be more concrete.

I know what you're talking about. But it just isn't practical. It isn't just turning off inlining - the types of the function parameters can reference private types or types in imported modules. So can parameter default values. So can const declarations. Templates too. There are two solutions you can use. Both involve putting your interfaces in a separate module. One means making a module just for your factory stub: ISearchEngine createSearchEngine (char[] someAttributes); Another is to use the COM approach of having a magic cookie (GUIDs serve as the magic cookie for COM) for each class, and pass that cookie to a generic factory function that is itself an interface member.
Dec 31 2005
next sibling parent Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:
 "Kris" <fu bar.com> wrote in message news:dp56kj$gc8$1 digitaldaemon.com...
 This setup does the right thing from an implementation-hiding aspect. It's 
 what's been asked for many times in the past. Note that only public 
 decalarations have been exposed. There's no class in there at all, just to 
 make it as simple as possible. Unfortunately, I can't show the -H version. 
 However, it's not much of a stretch to map the above onto Derek's original 
 example, and assert that (for example) the implementation for both the 
 factory method, and the static ctor() would be exposed (along with any 
 other method considered to be inline-able) That would be no good, Walter. 
 If it were the case, we'd have to ship all those proprietary imports too, 
 along with everything they referenced ~ ad infinitum.

 I do wish I had a working compiler so this could be more concrete.

I know what you're talking about. But it just isn't practical. It isn't just turning off inlining - the types of the function parameters can reference private types or types in imported modules. So can parameter default values. So can const declarations. Templates too.

Ack! I forgot about this. By the way Kris, I found a good description of the pimpl idiom here: http://www.gotw.ca/gotw/024.htm http://www.gotw.ca/gotw/028.htm
 There are two solutions you can use. Both involve putting your interfaces in 
 a separate module. One means making a module just for your factory stub:
     ISearchEngine createSearchEngine (char[] someAttributes);
 Another is to use the COM approach of having a magic cookie (GUIDs serve as 
 the magic cookie for COM) for each class, and pass that cookie to a generic 
 factory function that is itself an interface member.

Aye. See above :-) Sean
Dec 31 2005
prev sibling parent reply "Kris" <fu bar.com> writes:
Okay. I think there's an opportunity to clear this up via a few short 
questions. These refer back to the SearchEngine example posted previously 
(repeated below):

a) The example provided, where the SearchFactory.di file was hand-crafted? 
Will usage of that "di" module operate correctly?

b) You say below " it just isn't practical". Do you mean it is not 
technically feasible to mechanically extract the example SearchFactory.di 
from the given SearchFactory.d?


You'll hopefully note that example factory-module was explicitly written to 
avoid import references within all exposed types, has no templates, etc. 
Exactly the kind of thing a developer has to take responsibility over when 
decoupling usage from implementation (unless they explicitly wish to expose 
those also).

I have two more questions to follow up with, if you don't mind clearing this 
up?

- Kris


p.s. I'm reposting the example for the sake of clarity:

~~~~~~~~~~~~~~~~~~~
SearchInterfaces.d
~~~~~~~~~~~~~~~~~~~

module SearchInterfaces;

interface ISearchEngine
{
        ISearchItem[] lookup (char[] terms);
}

interface ISearchItem
{
        int relevancy();
        char[] retrievalUrl();
        char[] displaySnippet();
}


~~~~~~~~~~~~~~~~~~~
SearchFactory.d
~~~~~~~~~~~~~~~~~~~

module SearchFactory;


public import SearchInterfaces;

// private import a.bunch.of.proprietary.stuff;


// a factory for hooking up a client to the hidden implementation
public ISearchEngine createSearchEngine (char[] someAttributes)
{
        // potentially reference a.bunch.of.proprietary.stuff;
        return new XYX(someAttributes).
}


// do some proprietary setup
private static this()
{
        // potentially do a.bunch.of.proprietary.stuff;
}

~~~~~~~~~~~~~~~~~~~~~


That's the Factory implementation, which has a bunch of proprietary stuff in
it along with references to proprietary imports. To correctly decouple a
client from this implementation, one should provide another file (I'm
covering Walter's statements here, from a long time ago) along the following
lines:

~~~~~~~~~~~~~~~~~~~
SearchFactory.di
~~~~~~~~~~~~~~~~~~~

module SearchFactory;   // I believe this has to be the same name?

public import SearchInterfaces;

public ISearchEngine createSearchEngine (char[] someAttributes);

~~~~~~~~~~~~~~~~~~~




"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dp6hsf$1hdi$1 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dp56kj$gc8$1 digitaldaemon.com...
 This setup does the right thing from an implementation-hiding aspect. 
 It's what's been asked for many times in the past. Note that only public 
 decalarations have been exposed. There's no class in there at all, just 
 to make it as simple as possible. Unfortunately, I can't show the -H 
 version. However, it's not much of a stretch to map the above onto 
 Derek's original example, and assert that (for example) the 
 implementation for both the factory method, and the static ctor() would 
 be exposed (along with any other method considered to be inline-able) 
 That would be no good, Walter. If it were the case, we'd have to ship all 
 those proprietary imports too, along with everything they referenced ~ ad 
 infinitum.

 I do wish I had a working compiler so this could be more concrete.

I know what you're talking about. But it just isn't practical. It isn't just turning off inlining - the types of the function parameters can reference private types or types in imported modules. So can parameter default values. So can const declarations. Templates too. There are two solutions you can use. Both involve putting your interfaces in a separate module. One means making a module just for your factory stub: ISearchEngine createSearchEngine (char[] someAttributes); Another is to use the COM approach of having a magic cookie (GUIDs serve as the magic cookie for COM) for each class, and pass that cookie to a generic factory function that is itself an interface member.

Dec 31 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp6onr$1m6u$1 digitaldaemon.com...
 Okay. I think there's an opportunity to clear this up via a few short 
 questions. These refer back to the SearchEngine example posted previously 
 (repeated below):

 a) The example provided, where the SearchFactory.di file was hand-crafted? 
 Will usage of that "di" module operate correctly?

 b) You say below " it just isn't practical". Do you mean it is not 
 technically feasible to mechanically extract the example SearchFactory.di 
 from the given SearchFactory.d?

It isn't practical to determine that a declaration has *no* dependencies on any private class members or anything imported from an embedded import statement. It *is* practical to elide the bodies of functions that don't appear in templates, because there's no way for another declaration to look inside it, so -H does that. static this's are also elided, as nothing else can refer to them anyway. Think of it in these terms, and I think it'll become clear. -H can elide something *only* if it can guarantee there is nothing that can look inside it. Also, .di files have to be independent of changes in files the .di file imports. Thus, the import declarations have to remain, even if they are private imports. It is technically feasible to analyse the source code to the point where one can prove it depends on no private data, no private imports, etc., but that is a huge project, and would likely not yield much as few source files are written this way. Hence, I said it was impractical.
Dec 31 2005
parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dp7459$1stn$1 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dp6onr$1m6u$1 digitaldaemon.com...
 Okay. I think there's an opportunity to clear this up via a few short 
 questions. These refer back to the SearchEngine example posted previously 
 (repeated below):

 a) The example provided, where the SearchFactory.di file was 
 hand-crafted? Will usage of that "di" module operate correctly?

 b) You say below " it just isn't practical". Do you mean it is not 
 technically feasible to mechanically extract the example SearchFactory.di 
 from the given SearchFactory.d?

It isn't practical to determine that a declaration has *no* dependencies on any private class members or anything imported from an embedded import statement. It *is* practical to elide the bodies of functions that don't appear in templates, because there's no way for another declaration to look inside it, so -H does that. static this's are also elided, as nothing else can refer to them anyway.

Okay. And what about question (a) above, please?
Dec 31 2005
parent "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp76j5$1u9p$1 digitaldaemon.com...
 "Walter Bright" <newshound digitalmars.com> wrote in message 
 news:dp7459$1stn$1 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dp6onr$1m6u$1 digitaldaemon.com...
 Okay. I think there's an opportunity to clear this up via a few short 
 questions. These refer back to the SearchEngine example posted 
 previously (repeated below):

 a) The example provided, where the SearchFactory.di file was 
 hand-crafted? Will usage of that "di" module operate correctly?



It should work fine.
Jan 03 2006
prev sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Kris" <fu bar.com> wrote in message news:dp4oq5$6qt$1 digitaldaemon.com...
 I can point to one side-effect that actually does help: the compiler now 
 looks for "di" files, which means that hand-coded "implementation 
 bridging" files can (at last!) live side-by-side with the real 
 implementation module, eliminating some prior grief for a developer. 
 However, this conflicts with -H rather badly, and therefore is likely to 
 be an unintended and somewhat precarious benefit. Was that aspect 
 considered?

Actually, it's intended. You can automatically build them, hand build them, or hand edit the automatically built one.
Dec 30 2005
parent reply "Kris" <fu bar.com> writes:
"Walter Bright" <newshound digitalmars.com> wrote in message 
news:dp52vk$dqu$2 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dp4oq5$6qt$1 digitaldaemon.com...
 I can point to one side-effect that actually does help: the compiler now 
 looks for "di" files, which means that hand-coded "implementation 
 bridging" files can (at last!) live side-by-side with the real 
 implementation module, eliminating some prior grief for a developer. 
 However, this conflicts with -H rather badly, and therefore is likely to 
 be an unintended and somewhat precarious benefit. Was that aspect 
 considered?

Actually, it's intended. You can automatically build them, hand build them, or hand edit the automatically built one.

You can imagine the poor sod who carefully crafts his "di" file by hand (through necessity), only to find the "performance enhancing" aspects of the compiler trash his file at some later point. This ain't good.
Dec 30 2005
parent reply Sean Kelly <sean f4.ca> writes:
Kris wrote:
 "Walter Bright" <newshound digitalmars.com> wrote in message 
 news:dp52vk$dqu$2 digitaldaemon.com...
 "Kris" <fu bar.com> wrote in message 
 news:dp4oq5$6qt$1 digitaldaemon.com...
 I can point to one side-effect that actually does help: the compiler now 
 looks for "di" files, which means that hand-coded "implementation 
 bridging" files can (at last!) live side-by-side with the real 
 implementation module, eliminating some prior grief for a developer. 
 However, this conflicts with -H rather badly, and therefore is likely to 
 be an unintended and somewhat precarious benefit. Was that aspect 
 considered?

them, or hand edit the automatically built one.

You can imagine the poor sod who carefully crafts his "di" file by hand (through necessity), only to find the "performance enhancing" aspects of the compiler trash his file at some later point.

But don't you have to supply a -H command-line parameter to indicate that the header should be generated? I'm not sure it's fair to criticize DMD for what seems like user error. Sean
Dec 30 2005
parent "Kris" <fu bar.com> writes:
"Sean Kelly" <sean f4.ca> wrote...
 But don't you have to supply a -H command-line parameter to indicate that 
 the header should be generated?  I'm not sure it's fair to criticize DMD 
 for what seems like user error.

It's a precarious approach, which I noted in the OP. You've used Build before ... it's very easy to take a mis-step here, in all manner of ways.
Dec 30 2005
prev sibling parent reply James Dunne <james.jdunne gmail.com> writes:
For implementation hiding, just move the private members in classes to 
the end of the vtable, after all public and protected members (define 
this in the ABI and modify the reference compiler to guarantee this). 
Problem solved.

No more private members need to be included in the generated header 
files because their vtable offsets are always greater than vtable 
offsets of public/protected members and will have no push/pull affect on 
them if left out.  Simply ignoring them in the _header_ class creates no 
binary compatibility issues.

Const initializers; simply include them as well in the generated header 
classes for public/protected members to preserve correct offsets.

Templates... well that's another story.
Dec 30 2005
next sibling parent John Reimer <terminal.node gmail.com> writes:
James Dunne wrote:
 For implementation hiding, just move the private members in classes to 
 the end of the vtable, after all public and protected members (define 
 this in the ABI and modify the reference compiler to guarantee this). 
 Problem solved.
 
 No more private members need to be included in the generated header 
 files because their vtable offsets are always greater than vtable 
 offsets of public/protected members and will have no push/pull affect on 
 them if left out.  Simply ignoring them in the _header_ class creates no 
 binary compatibility issues.
 
 Const initializers; simply include them as well in the generated header 
 classes for public/protected members to preserve correct offsets.
 

I was wondering about this!
 Templates... well that's another story.

I guess it is another story. Maybe templates should just stay in there own D file and be processed the old fashioned way. Oh boy, look at us. We're all still trying to solve the implementation-hiding issue, insatiable creatures we are. :) -JJR
Dec 30 2005
prev sibling next sibling parent reply "Derek Parnell" <derek psych.ward> writes:
On Sat, 31 Dec 2005 18:03:46 +1100, James Dunne <james.jdunne gmail.com>  
wrote:

 For implementation hiding, just move the private members in classes to  
 the end of the vtable, after all public and protected members (define  
 this in the ABI and modify the reference compiler to guarantee this).  
 Problem solved.

 No more private members need to be included in the generated header  
 files because their vtable offsets are always greater than vtable  
 offsets of public/protected members and will have no push/pull affect on  
 them if left out.  Simply ignoring them in the _header_ class creates no  
 binary compatibility issues.

 Const initializers; simply include them as well in the generated header  
 classes for public/protected members to preserve correct offsets.

 Templates... well that's another story.

I'm sure this is all true, but wouldn't it be nice if we had some automated way of achieving this end, rather than relying on hand crafted code. -- Derek Parnell Melbourne, Australia
Dec 31 2005
next sibling parent John Reimer <terminal.node gmail.com> writes:
Derek Parnell wrote:
 On Sat, 31 Dec 2005 18:03:46 +1100, James Dunne <james.jdunne gmail.com> 
 wrote:
 
 For implementation hiding, just move the private members in classes to 
 the end of the vtable, after all public and protected members (define 
 this in the ABI and modify the reference compiler to guarantee this). 
 Problem solved.

 No more private members need to be included in the generated header 
 files because their vtable offsets are always greater than vtable 
 offsets of public/protected members and will have no push/pull affect 
 on them if left out.  Simply ignoring them in the _header_ class 
 creates no binary compatibility issues.

 Const initializers; simply include them as well in the generated 
 header classes for public/protected members to preserve correct offsets.

 Templates... well that's another story.

I'm sure this is all true, but wouldn't it be nice if we had some automated way of achieving this end, rather than relying on hand crafted code. --Derek Parnell Melbourne, Australia

I think that was the idea he was trying to get across: it's not impossible to do.
Dec 31 2005
prev sibling parent James Dunne <james.jdunne gmail.com> writes:
Derek Parnell wrote:
 On Sat, 31 Dec 2005 18:03:46 +1100, James Dunne 
 <james.jdunne gmail.com>  wrote:
 
 For implementation hiding, just move the private members in classes 
 to  the end of the vtable, after all public and protected members 
 (define  this in the ABI and modify the reference compiler to 
 guarantee this).  Problem solved.

 No more private members need to be included in the generated header  
 files because their vtable offsets are always greater than vtable  
 offsets of public/protected members and will have no push/pull affect 
 on  them if left out.  Simply ignoring them in the _header_ class 
 creates no  binary compatibility issues.

 Const initializers; simply include them as well in the generated 
 header  classes for public/protected members to preserve correct offsets.

 Templates... well that's another story.

I'm sure this is all true, but wouldn't it be nice if we had some automated way of achieving this end, rather than relying on hand crafted code.

That's what I was saying... a compiler change. It would be silly to require all D programmers to define their private members at the end of their classes.
Dec 31 2005
prev sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"James Dunne" <james.jdunne gmail.com> wrote in message 
news:dp5agf$ioj$1 digitaldaemon.com...
 For implementation hiding, just move the private members in classes to the 
 end of the vtable, after all public and protected members (define this in 
 the ABI and modify the reference compiler to guarantee this). Problem 
 solved.

That works until you try to derive from it. <g>
 Const initializers; simply include them as well in the generated header 
 classes for public/protected members to preserve correct offsets.

? Const initializers may refer to private declarations.
 Templates... well that's another story.

OOP is implementation hiding, templates are the antithesis of implementation hiding. In C++, exported templates were created to solve this problem, but it's pretty obvious to me that it does no such thing. I haven't been too popular in some circles for pointing this out <g>.
Dec 31 2005
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Walter Bright wrote:
 "James Dunne" <james.jdunne gmail.com> wrote in message 
 
 Templates... well that's another story.

OOP is implementation hiding, templates are the antithesis of implementation hiding. In C++, exported templates were created to solve this problem, but it's pretty obvious to me that it does no such thing. I haven't been too popular in some circles for pointing this out <g>.

There's been a lot of confusion about the real purpose of "export" in C++, much like the -H option here ;-) All the feature really provides is a way to speed compilation--implementation hiding is not really possible (as you've pointed out). Whether potential benefit of "export" is worth the development cost is an issue of much contention. The folks at EDG argued against the feature before it was made standard, but now they're arguing for it... possibly because they've spent so much time implementing it. I don't think Microsoft has any plans of implementing "export," though I haven't heard anything from other vendors. Sean
Dec 31 2005
parent "Walter Bright" <newshound digitalmars.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message 
news:dp6j8o$1ibk$1 digitaldaemon.com...
 There's been a lot of confusion about the real purpose of "export" in C++, 
 much like the -H option here ;-)  All the feature really provides is a way 
 to speed compilation--implementation hiding is not really possible (as 
 you've pointed out).

I don't believe it speeds up compilation either. None of the EDG compilers have been willing to take up my challenge on DMC++ with precompiled headers vs EDG and exported templates <g>.
 Whether potential benefit of "export" is worth the development cost is an 
 issue of much contention.

The people who argue it is worth it are either the ones for which it is a sunk cost or are not the ones who'll have to pay for it. Although I've pointed out that the ones who think they don't have to pay for it will actually have to pay for it, in terms that they won't get other features/support they probably need much more.
 The folks at EDG argued against the feature before it was made standard, 
 but now they're arguing for it... possibly because they've spent so much 
 time implementing it.

For EDG, it's a sunk cost. Their cost for the feature going forward is minimal, so once it is a sunk cost, the cost/benefit ratio suddenly becomes very favorable. It's in EDG's economic best interest at this point to strongly support export.
 I don't think Microsoft has any plans of implementing "export," though I 
 haven't heard anything from other vendors.

I haven't heard anyone else planning to implement it, either.
Dec 31 2005
prev sibling parent reply James Dunne <james.jdunne gmail.com> writes:
Walter Bright wrote:
 "James Dunne" <james.jdunne gmail.com> wrote in message 
 news:dp5agf$ioj$1 digitaldaemon.com...
 
For implementation hiding, just move the private members in classes to the 
end of the vtable, after all public and protected members (define this in 
the ABI and modify the reference compiler to guarantee this). Problem 
solved.

That works until you try to derive from it. <g>

*thinking...* Ow. Well, who says the binary layout of derived classes must be equivalent? Isn't this just an implementation detail that most compilers assume is true in order to gain speed and ease of implementation for virtual/overridden methods? I'll be thinking this one out for a while ...
Const initializers; simply include them as well in the generated header 
classes for public/protected members to preserve correct offsets.

? Const initializers may refer to private declarations.

So there's the catcher... I didn't know that, thanks! Well, why not const-fold out that private declaration into the initializer? Oh right... again with the deriving... I always forget about the deriving...
 
Templates... well that's another story.

OOP is implementation hiding, templates are the antithesis of implementation hiding. In C++, exported templates were created to solve this problem, but it's pretty obvious to me that it does no such thing. I haven't been too popular in some circles for pointing this out <g>.

Anyone wishing to hide their implementation of templates should be reconsidering their worth to society. =)
Dec 31 2005
parent "Kris" <fu bar.com> writes:
"James Dunne" <james.jdunne gmail.com> wrote in

 Anyone wishing to hide their implementation of templates should be 
 reconsidering their worth to society. =)

Amen.
Dec 31 2005
prev sibling parent reply Carlos Santander <csantander619 gmail.com> writes:
John Reimer escribió:
 
 Okay, now I see why there is so much confusion going on with the 
 addition of this feature.  There are two problems to be solved: one is 
 the issue of implementation hiding; the other is the issue of compiler 
 performance.  I think most of us mistook the di file generator to be a 
 solution for implementation hiding for closed source projects. Obviously 
 we got the wrong idea about the whole thing.
 
 If di files serve the same purpose as "precompiled headers," obviously 
 there is no consideration whatsoever for implementation hiding (and 
 there never was any such intention).
 
 Good to know.  Personally, I was less interested in compile performance 
 and more interested in implementation hiding.  The -H flag, therefore, 
 will not be of much use to me.
 
 -JJR

Same here. I mean, I also agree with all of you guys that -H should do something different. BTW, IIRC, digc did something like that. Does anybody know if it still compiles? Did it really do that? -- Carlos Santander Bernal
Dec 30 2005
parent John Reimer <terminal.node gmail.com> writes:
Carlos Santander wrote:
 John Reimer escribió:
 Okay, now I see why there is so much confusion going on with the 
 addition of this feature.  There are two problems to be solved: one is 
 the issue of implementation hiding; the other is the issue of compiler 
 performance.  I think most of us mistook the di file generator to be a 
 solution for implementation hiding for closed source projects. 
 Obviously we got the wrong idea about the whole thing.

 If di files serve the same purpose as "precompiled headers," obviously 
 there is no consideration whatsoever for implementation hiding (and 
 there never was any such intention).

 Good to know.  Personally, I was less interested in compile 
 performance and more interested in implementation hiding.  The -H 
 flag, therefore, will not be of much use to me.

 -JJR

Same here. I mean, I also agree with all of you guys that -H should do something different. BTW, IIRC, digc did something like that. Does anybody know if it still compiles? Did it really do that?

I think digc was updated several months ago by Burton to do a whole lot more than it used to do. It might be worth checking out. I distinctly recall that it received very little attention when he announced the update, which was a surprise to me. I haven't looked into it much myself, though. -JJR
Dec 30 2005
prev sibling next sibling parent reply bobef <bobef lessequal.com> writes:
Walter Bright wrote:
 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!
 
 http://www.digitalmars.com/d/changelog.html
 
 

I play OpenTTD all night and in the morning see what happens :) Seriously. I won't participate in the dispute because I am not 100% sure I fully understand it (I am weak with all these programming terms :), but I wish to support Walter in some way. As much as I understand the new feature helps speeding up the compiler but it isn't hiding all the code wich comes to be a problem for people wanting close source projects. From the posts I almost thought nobody feels the slow compilition a problem and it is the "people vs Walter". I want to say that this was exactly what I needed, because the compilation speed is a big issue to me. Once I started a thread about this. (Now I am shooting in the dark sorry if I am wrong) It seems that many people are using the build tool (which I've never used but I guess it works this way) or even if they are not they pass all the source files to the compiler. In this case the speed is optimal but if you compile each file separately it becomes like 15 times slower (for my project). And I do it because I have more control over the process this way. So I just wanted to support Walter and say that this is a very useful feature for me. Thank you, Walter (with much help from Dave Fladebo ;). P.S. I didn't undestand how you control if the compiler is importing .d or .di . I looked in the docs but I didn't find any info on that. I mean if they have the same name?
Dec 30 2005
next sibling parent reply "Derek Parnell" <derek psych.ward> writes:
On Fri, 30 Dec 2005 21:10:52 +1100, bobef <bobef lessequal.com> wrote:
 P.S. I didn't undestand how you control if the compiler is importing .d  
 or .di . I looked in the docs but I didn't find any info on that. I mean  
 if they have the same name?

If you code "import xyz" the compiler looks first for xyz.d, and if it can't find that it tries to find xyz.di and if that's not found it errors. -- Derek Parnell Melbourne, Australia
Dec 30 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"Derek Parnell" <derek psych.ward> wrote in message 
news:op.s2le7jga6b8z09 ginger.vic.bigpond.net.au...
 On Fri, 30 Dec 2005 21:10:52 +1100, bobef <bobef lessequal.com> wrote:
 P.S. I didn't undestand how you control if the compiler is importing .d 
 or .di . I looked in the docs but I didn't find any info on that. I mean 
 if they have the same name?

If you code "import xyz" the compiler looks first for xyz.d, and if it can't find that it tries to find xyz.di and if that's not found it errors.

No, first it looks for .di, then .d.
Dec 30 2005
parent "Derek Parnell" <derek psych.ward> writes:
On Sat, 31 Dec 2005 10:28:10 +1100, Walter Bright  
<newshound digitalmars.com> wrote:

 "Derek Parnell" <derek psych.ward> wrote in message
 news:op.s2le7jga6b8z09 ginger.vic.bigpond.net.au...
 On Fri, 30 Dec 2005 21:10:52 +1100, bobef <bobef lessequal.com> wrote:
 P.S. I didn't undestand how you control if the compiler is importing .d
 or .di . I looked in the docs but I didn't find any info on that. I  
 mean
 if they have the same name?

If you code "import xyz" the compiler looks first for xyz.d, and if it can't find that it tries to find xyz.di and if that's not found it errors.

No, first it looks for .di, then .d.

Brillilant! I think I can use that behaviour in the macro preprocessor coming soon to Build. -- Derek Parnell Melbourne, Australia
Dec 31 2005
prev sibling next sibling parent reply Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
bobef wrote:
 Walter Bright wrote:
 
 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!

 http://www.digitalmars.com/d/changelog.html

I play OpenTTD all night and in the morning see what happens :)

Wow! It is one of the *best* games I ever played. It is great to see there are more fans.
Dec 30 2005
next sibling parent John Reimer <terminal.node gmail.com> writes:
Ivan Senji wrote:
 bobef wrote:
 Walter Bright wrote:

 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!

 http://www.digitalmars.com/d/changelog.html

I play OpenTTD all night and in the morning see what happens :)

Wow! It is one of the *best* games I ever played. It is great to see there are more fans.

Hmmm... I've never tried it before. I'll have to take a look. -JJR
Dec 30 2005
prev sibling parent reply John Reimer <terminal.node gmail.com> writes:
Ivan Senji wrote:
 bobef wrote:
 Walter Bright wrote:

 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!

 http://www.digitalmars.com/d/changelog.html

I play OpenTTD all night and in the morning see what happens :)

Wow! It is one of the *best* games I ever played. It is great to see there are more fans.

Woah! I didn't realize that I needed the original game graphics in order to make this work. So much for that. -JJR
Dec 30 2005
parent Ivan Senji <ivan.senji_REMOVE_ _THIS__gmail.com> writes:
John Reimer wrote:
 Ivan Senji wrote:
 
 bobef wrote:

 Walter Bright wrote:

 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!

 http://www.digitalmars.com/d/changelog.html

I play OpenTTD all night and in the morning see what happens :)

Wow! It is one of the *best* games I ever played. It is great to see there are more fans.

Woah! I didn't realize that I needed the original game graphics in order to make this work. So much for that.

You can find patched original version of the game, it is practically the same as OpenTTD and has original graphics. But you have to be warned that some people really don't like this type of games. I spent countless hours playing it, but some people just don't get it. :)
Dec 30 2005
prev sibling next sibling parent reply J C Calvarese <technocrat7 gmail.com> writes:
In article <dp3132$1r9i$1 digitaldaemon.com>, bobef says...
I play OpenTTD all night and in the morning see what happens :)
Seriously. I won't participate in the dispute because I am not 100% sure 
I fully understand it (I am weak with all these programming terms :), 

Really, I think it's been more of a misunderstanding than anything else. When Walter announced that DMD 0.142 could generate header, a few people started salivating at the thought of having the ability to hide the implementation of files. It's been request quite a few times, and I'd rather be able to tell them about "the H switch" than have to teach someone how to do it manually. But unfortunately, when these eager D citizens tried out the new compiler function they realized something was terribly wrong. The compiler wouldn't hide things that one would expect it to hide. "Hey, Walter, I think it's broken," they say. "No," says Walter. "That's how it's supposed to work. Unless it crashes -- I'll fix the things that make it crash." And I'm not saying that what it does isn't useful. Obviously, it must useful to Dave or he wouldn't have bothered implemented it, and I can see how it could speed compilation of large projects. I think the ability to hide the implementation (as much as possible) would be of a greater benefit to the D community, but I don't see a reason why we couldn't eventually have both. So it's more of a case of openning up a present that you think is going to be a cool DVD, and it ends up being a book. It might be a very good book, but you were hoping for a movie. (And I do like to read, too; I'm not illiterate or anything.) ;)
but I wish to support Walter in some way. As much as I understand the 
new feature helps speeding up the compiler but it isn't hiding all the 
code wich comes to be a problem for people wanting close source 
projects. From the posts I almost thought nobody feels the slow 
compilition a problem and it is the "people vs Walter". I want to say 
that this was exactly what I needed, because the compilation speed is a 
big issue to me. Once I started a thread about this. (Now I am shooting 
in the dark sorry if I am wrong) It seems that many people are using the 
build tool (which I've never used but I guess it works this way) or even 
if they are not they pass all the source files to the compiler. In this 
case the speed is optimal but if you compile each file separately it 
becomes like 15 times slower (for my project). And I do it because I 
have more control over the process this way.

So I just wanted to support Walter and say that this is a very useful 
feature for me. Thank you, Walter (with much help from Dave Fladebo ;).

I'm not at all against Walter adding this feature to the compiler (and I don't think anyone else really is against it either). I just think he should have called it something else in the announcement. Like "header generator for faster compiling". (Of course, everyone is much better of naming stuff than I am, so that's not a very good suggestion.) Oh, well. What's done is done. So thanks Walter and Dave. Good job. It's not what I expected, but maybe it's just what D needs. ;) jcc7
Dec 30 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"J C Calvarese" <technocrat7 gmail.com> wrote in message 
news:dp3vqv$2jkt$1 digitaldaemon.com...
 But unfortunately, when these eager D citizens tried out the new compiler
 function they realized something was terribly wrong. The compiler wouldn't 
 hide
 things that one would expect it to hide. "Hey, Walter, I think it's 
 broken,"
 they say.

 "No," says Walter. "That's how it's supposed to work. Unless it crashes --  
 I'll
 fix the things that make it crash."

It winds up putting the same things in the 'header' that you have to put in in C++, for the same reasons.
Dec 30 2005
parent J C Calvarese <technocrat7 gmail.com> writes:
In article <dp4hdq$31jl$3 digitaldaemon.com>, Walter Bright says...
"J C Calvarese" <technocrat7 gmail.com> wrote in message 
news:dp3vqv$2jkt$1 digitaldaemon.com...
 But unfortunately, when these eager D citizens tried out the new compiler
 function they realized something was terribly wrong. The compiler wouldn't 
 hide
 things that one would expect it to hide. "Hey, Walter, I think it's 
 broken,"
 they say.

 "No," says Walter. "That's how it's supposed to work. Unless it crashes --  
 I'll
 fix the things that make it crash."

It winds up putting the same things in the 'header' that you have to put in in C++, for the same reasons.

After reading some of the most recent posts, I think I understand better than I did before what some of these issues boil down to. Apparently, implementation-hiding is _a lot_ more complicated than I realized. Part of why I thought that it wouldn't be that complicated is because dig's strip.d is a pretty short program. I didn't really test it, but I thought the dig's strip.d worked pretty good. Now, I'm pretty sure it could work only because it wasn't thoroughly tested with complicated enough code to make it choke. jcc7
Jan 01 2006
prev sibling parent "Walter Bright" <newshound digitalmars.com> writes:
"bobef" <bobef lessequal.com> wrote in message 
news:dp3132$1r9i$1 digitaldaemon.com...
 So I just wanted to support Walter and say that this is a very useful 
 feature for me. Thank you, Walter (with much help from Dave Fladebo ;).

You're welcome!
 P.S. I didn't undestand how you control if the compiler is importing .d or 
 .di . I looked in the docs but I didn't find any info on that. I mean if 
 they have the same name?

It looks at each directory in the import path, first for a .di file, then a .d file.
Dec 30 2005
prev sibling parent Bruno Medeiros <I.AM FAIAL.com> writes:
Walter Bright wrote:
 This incorporates a new 'header' generator capability, written by Dave
 Fladebo, now working!
 
 http://www.digitalmars.com/d/changelog.html
 
 

Jeez, what a mess of a thread. Before of all, as for this pre-parse header feature, since it is (primarily?) a performance feature, is it planned that in the future (after debugging) this option will be enabled by default? Now, what is the appropriate way to create and use in D an external library, in terms of module importation? Is it to use the whole library source and import it's modules (but not compile them of course)?, Or to hand-craft header modules? But what if I want to use different versions of said library with the same aplication *executable* (say, like updating a buggy DLL)? This makes the first option (use full source) inviable, right? (because the the bugfix may be in a inlined function?) Thus, if hand-crafting is the only solution, wouldn't it be useful to have a tool that would automatically generate this interface(the set of headers)? However, what is the interface of a module and what is it's implemention? I think that it is in this question that many of the confusion of this thread originated. For Walter, this interface is everything that might be need by the importing module, even on a binary level, so this includes privates and inlinable function bodies. This, is the standard C++ point of view and pratice, as Walter pointed out and exemplified. However, since in many cases the conceptual interface may be smaller than this standard interface, some people "sugested" that the generated interface(header) should be closer to this conceptual interface, particularly: it should not contain privates and function bodies. However, for this to be done(implemented), it has to be *correct* and *clear* on how it would work. People said that privates shouldn't appear on the generated headers, but "private" and protection attributes in general are (again I say this) not made for code hiding, but just for protection. And so, when using them for information hiding (like removing private code from generated headers), it doesn't work, as Walter pointed out in many exemples (used internally, problems with the binary interface, etc.). So if one wants to complain about not having this feature implemented, then it should first of all be well tought out and defined how the feature works. (and unfortunately that didn't happen) That means extending and clarifying the semantics of private for such purpose. For instance when a public function has a private parameter. (I wonder what happens in Java or C# , but I'm not yet back home with my PC, so I can't try it out)
Jan 01 2006