www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compiler as dll

reply bearophile <bearophileHUGS lycos.com> writes:
In C# 4.0 the VM becomes a module of the standard lib, so it can be used from
normal C# code, to do various things.

While writing a genetic programming program (and in some other situations) I
find it useful to have an eval() function, like in Lisp/Scheme and most
scripting languages.

I think such functionalities may be added to DMD and LDC too:
- 99% of the compiler can be moved into a DLL (or a shared dynamic lib of some
kind) and the DMD/LDC executable can then become very little, it can load such
dynamic lib, and it has just to manage the I/O from/to disk and the command
line arguments, ad little else.
- By default such dynamic lib isn't loaded by D programs, so the size of D
programs is unchanged (and they don't need such dll to run).
- A module can be added to the std lib (Tango and Phobos) that loads such dll
and offers a function like eval() that accepts in input a string of code (or an
AST too, if you want) and compiles/runs it.

Bye,
bearophile
Jan 27 2009
next sibling parent Christopher Wright <dhasenan gmail.com> writes:
bearophile wrote:
 In C# 4.0 the VM becomes a module of the standard lib, so it can be used from
normal C# code, to do various things.
 
 While writing a genetic programming program (and in some other situations) I
find it useful to have an eval() function, like in Lisp/Scheme and most
scripting languages.
 
 I think such functionalities may be added to DMD and LDC too:
 - 99% of the compiler can be moved into a DLL (or a shared dynamic lib of some
kind) and the DMD/LDC executable can then become very little, it can load such
dynamic lib, and it has just to manage the I/O from/to disk and the command
line arguments, ad little else.
 - By default such dynamic lib isn't loaded by D programs, so the size of D
programs is unchanged (and they don't need such dll to run).
 - A module can be added to the std lib (Tango and Phobos) that loads such dll
and offers a function like eval() that accepts in input a string of code (or an
AST too, if you want) and compiles/runs it.

I think Walter would take Oren Eini's approach to this: send me a patch for it.
 Bye,
 bearophile

Jan 27 2009
prev sibling next sibling parent "Saaa" <empty needmail.com> writes:
Where and how do you use genetic programming ?? (just interested : )


 In C# 4.0 the VM becomes a module of the standard lib, so it can be used 
 from normal C# code, to do various things.

 While writing a genetic programming program (and in some other situations) 
 I find it useful to have an eval() function, like in Lisp/Scheme and most 
 scripting languages.

 I think such functionalities may be added to DMD and LDC too:
 - 99% of the compiler can be moved into a DLL (or a shared dynamic lib of 
 some kind) and the DMD/LDC executable can then become very little, it can 
 load such dynamic lib, and it has just to manage the I/O from/to disk and 
 the command line arguments, ad little else.
 - By default such dynamic lib isn't loaded by D programs, so the size of D 
 programs is unchanged (and they don't need such dll to run).
 - A module can be added to the std lib (Tango and Phobos) that loads such 
 dll and offers a function like eval() that accepts in input a string of 
 code (or an AST too, if you want) and compiles/runs it.

 Bye,
 bearophile 

Jan 27 2009
prev sibling next sibling parent reply Sandeep Kakarlapudi <sandeep.iitkgpspammenot gmail.com> writes:
I have been wishing for something similar though for different reasons. Often I
have code (in C++) that is executed for a large amount of data like in stream
programming and becomes a major hotspot for the application. Think software
graphics pipelines. Further, this code has lots of branches and since its
executed over many elements they are essentially branches in inner-loops. The
values over which they branch are unknown till runtime and the number of
combinations of the values can be very very high. Here are the basic attempts
to solve that I'm aware of and the problems with each:

1) Programmer codes for each combination of conditions - Takes a lot of time
and is hard to maintain. 

2) Use metaprogramming to generate all combinations at compile time. I usually
go for this using C++ templates. This can bloat the binary size very quickly
when the number of values are many. 

3) Generate the code at runtime. Often the values don't change too often and
hence generating the code at runtime would help. But here the   programmer
often has dive into assembly. IIRC C# libraries allow you to target its
bytecode. 

4) With the philosophy of Life Long Program Optimization, the code is
monitored/ traced and automatically optimized. Even if such a language targets
some intermediate bytecode, there can be cases where it might significantly
outperform native code by using this approach. Might not be suitable for a
language that targets native code?? Not suited for systems programming. 

5) Perhaps with some support in the language/library the application can invoke
the optimizer and give the know values and the optimizer can create an
optimized version a function for those values. Is this feasible? 

In summary, what I'm asking for is provision for the programmer to direct code
specialization at runtime. This becomes a subset of bearophile's request so I
would also like to see a more generic and standard(!) way of runtime compiling
/ optimization of high level code. 

I also wonder if any of the std library functionality can be better implemented
given such runtime code compilation and optimization features.

Sandeep 

bearophile Wrote:

 In C# 4.0 the VM becomes a module of the standard lib, so it can be used from
normal C# code, to do various things.
 
 While writing a genetic programming program (and in some other situations) I
find it useful to have an eval() function, like in Lisp/Scheme and most
scripting languages.
 
 I think such functionalities may be added to DMD and LDC too:
 - 99% of the compiler can be moved into a DLL (or a shared dynamic lib of some
kind) and the DMD/LDC executable can then become very little, it can load such
dynamic lib, and it has just to manage the I/O from/to disk and the command
line arguments, ad little else.
 - By default such dynamic lib isn't loaded by D programs, so the size of D
programs is unchanged (and they don't need such dll to run).
 - A module can be added to the std lib (Tango and Phobos) that loads such dll
and offers a function like eval() that accepts in input a string of code (or an
AST too, if you want) and compiles/runs it.
 
 Bye,
 bearophile

Jan 27 2009
parent reply BCS <ao pathlink.com> writes:
Reply to Sandeep,

 I have been wishing for something similar though for different
 reasons. Often I have code (in C++) that is executed for a large
 amount of data like in stream programming and becomes a major hotspot
 for the application. Think software graphics pipelines. Further, this
 code has lots of branches and since its executed over many elements
 they are essentially branches in inner-loops. The values over which
 they branch are unknown till runtime and the number of combinations of
 the values can be very very high. Here are the basic attempts to solve
 that I'm aware of and the problems with each:
 

One option you could use under GCC would be to build all the inner control structures out of gotos and use goto variables that are set outside the loop. I'm not sure how dynamic gotos stack up with conditional jumps but you would still avoid the tests.
 
 Sandeep
 

Jan 27 2009
parent Sandeep Kakarlapudi <sandeep.iitkgpspammenot gmail.com> writes:
BCS Wrote:

 Reply to Sandeep,
 
 I have been wishing for something similar though for different
 reasons. Often I have code (in C++) that is executed for a large
 amount of data like in stream programming and becomes a major hotspot
 for the application. Think software graphics pipelines. Further, this
 code has lots of branches and since its executed over many elements
 they are essentially branches in inner-loops. The values over which
 they branch are unknown till runtime and the number of combinations of
 the values can be very very high. Here are the basic attempts to solve
 that I'm aware of and the problems with each:
 

One option you could use under GCC would be to build all the inner control structures out of gotos and use goto variables that are set outside the loop. I'm not sure how dynamic gotos stack up with conditional jumps but you would still avoid the tests.

Nice idea. While it still won't have optimal peformance, it definitely is another option to use. It gets rid of code bloat, but it comes at the cost of readability and hence maintainability. But I think none of these would come close to the having a run time optimizer to perform code specialization. Often the small latency in optimizing the code is well worth its result.
Jan 28 2009
prev sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
I've done the compiler-as-dll thing for the Digital Mars IDE. It has 
some problems, though. The biggest is it's another executable to test, 
doubling the testing process. It could be done as one executable by 
making a shell that calls the dll, but those are just inconvenient, 
harder to debug, and there's the old "dll hell" with versions.

Instead, what you can do is simply dude up command line arguments, spawn 
the command line compiler, and collect the result.
Jan 27 2009
next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Jan 27, 2009 at 6:44 PM, Walter Bright
<newshound1 digitalmars.com> wrote:
 Instead, what you can do is simply dude up command line arguments, spawn the
 command line compiler, and collect the result.

Process startup incurs a very large performance hit. Yes, you wouldn't want to be calling even a library compiler in a tight loop, but still, it precludes doing at least semi-realtime things. (though the D compiler is so fast that it doesn't matter much anyway - see h3r3tic's nucled) Another thing having it as a library enables is much simpler IDE integration, as you probably know already. Furthermore, if the compiler's interface is made flexible enough, you can also do some really interesting things - have callbacks for imports which will automatically fetch and install libraries that you don't already have, for instance. Things like DSSS would just become different frontends for the compiler.
Jan 27 2009
prev sibling next sibling parent reply BCS <none anon.com> writes:
Hello Walter,

 
 Instead, what you can do is simply dude up command line arguments,
 spawn the command line compiler, and collect the result.
 

The one main thing I see not working there is memory-to-memory compiles. I'd love to be able to build a function as a string, call the compiler and get back a function pointer.
Jan 27 2009
parent reply Benji Smith <dlanguage benjismith.net> writes:
BCS wrote:
 Hello Walter,
 
 Instead, what you can do is simply dude up command line arguments,
 spawn the command line compiler, and collect the result.

The one main thing I see not working there is memory-to-memory compiles. I'd love to be able to build a function as a string, call the compiler and get back a function pointer.

I think also, with a compiler-as-dll, it'd have separate modules for lexing, parsing, optimizing, code-generation, and linking. As a user of that compiler DLL, I might like to write my own AST visitor, wrapping all function calls (or scope blocks) with tracing statements before sending them into the rest of the pipeline. Those are the kinds of things that I think would be especially cool with a CompilerServices module in the standard library. Also, consider this: someone could implement AST macros as a library! --benji
Jan 27 2009
parent reply Yigal Chripun <yigal100 gmail.com> writes:
Benji Smith wrote:
 BCS wrote:
 Hello Walter,

 Instead, what you can do is simply dude up command line arguments,
 spawn the command line compiler, and collect the result.

The one main thing I see not working there is memory-to-memory compiles. I'd love to be able to build a function as a string, call the compiler and get back a function pointer.

I think also, with a compiler-as-dll, it'd have separate modules for lexing, parsing, optimizing, code-generation, and linking. As a user of that compiler DLL, I might like to write my own AST visitor, wrapping all function calls (or scope blocks) with tracing statements before sending them into the rest of the pipeline. Those are the kinds of things that I think would be especially cool with a CompilerServices module in the standard library. Also, consider this: someone could implement AST macros as a library! --benji

You've just described the design of clang - the new frontend for llvm for compiling c/c++. there are many tools used during development which need different subsets of functionality of the compiler that will benefit from such a design: IDE - the IDE needs to use the lexing/parsing/semantic phases for showing you errors on the fly, it's integrated build system needs to know about dependencies, and it'll use the optimizer/code-gen phases to build your projects. lint tools - need the lexing/parsing/semantic phases. stand alone build tool (like rebuild, bud, etc) - needs the lexer/parser for automatically resolving dependencies. doc system (built in IDE/ or stand alone) needs the semantic phases. the compiler itself could (and should) use it - currently there's a built in interpreter for CTFE which is limited to a subset of D. instead of implementing a limited interpreter in addition to the compiler, the compiler libs can easily be utilized for a JIT compiler which will compile CTFE and run them during compilation using the same code that is used to compile regular code.
Jan 27 2009
parent Sandeep Kakarlapudi <sandeep.iitkgpspammenot gmail.com> writes:
Yigal Chripun Wrote:

 the compiler itself could (and should) use it - currently there's a 
 built in interpreter for CTFE which is limited to a subset of D.
 
 instead of implementing a limited interpreter in addition to the 
 compiler, the compiler libs can easily be utilized for a JIT compiler 
 which will compile CTFE and run them during compilation using the same 
 code that is used to compile regular code.

I had the exact same thoughts a couple of days ago - but I remembered that floating point code *might* cause problems either because of precision modes or cross compiling. However going in this direction seems to open up more possibilities.
Jan 28 2009
prev sibling parent reply Yigal Chripun <yigal100 gmail.com> writes:
Walter Bright wrote:
 I've done the compiler-as-dll thing for the Digital Mars IDE. It has
 some problems, though. The biggest is it's another executable to test,
 doubling the testing process. It could be done as one executable by
 making a shell that calls the dll, but those are just inconvenient,
 harder to debug, and there's the old "dll hell" with versions.

 Instead, what you can do is simply dude up command line arguments, spawn
 the command line compiler, and collect the result.

here's a thought: use DDL instead. no more "dll hell" :)
Jan 27 2009
next sibling parent reply John Reimer <terminal.node gmail.com> writes:
Hello Yigal,

 Walter Bright wrote:
 
 I've done the compiler-as-dll thing for the Digital Mars IDE. It has
 some problems, though. The biggest is it's another executable to
 test, doubling the testing process. It could be done as one
 executable by making a shell that calls the dll, but those are just
 inconvenient, harder to debug, and there's the old "dll hell" with
 versions.
 
 Instead, what you can do is simply dude up command line arguments,
 spawn the command line compiler, and collect the result.
 


There's one thing ddl doesn't do that dll's/so's do. People seem to misunderstand this aspect of it, as I did for awhile (until recently). ddl does not work for memory sharing like normal dll's, where multiple applications have access to a single dll at runtime. It appears that such support would be quite difficult to implement and moves in the direction of operating system features. It does do runtime linking, however, which is extremely useful for certain situations... specifically any sort of application that needs a plugin architecture for D (ie.. it can link with libraries and object files at runtime) that is gc and exception friendly. Thanks to Tom S. for clarifying this to me. Whether or not this detail is significant to the idea of a compiler dll, I don't know. -JJR
Jan 28 2009
parent reply grauzone <none example.net> writes:
John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where multiple 
 applications have access to a single dll at runtime.  It appears that 
 such support would be quite difficult to implement and moves in the 
 direction of operating system features.

Couldn't this be achieved by simply mmap()-ing the file contents into memory? mmap() normally shared the memory pages with other processes. Of course, this wouldn't work if the code both isn't position independent, and needs to be relocated to a different base address. But that's also the case with operating system supported dynamic shared objects.
 It does do runtime linking, however, which is extremely useful for 
 certain situations... specifically any sort of application that needs a 
 plugin architecture for D (ie.. it can link with libraries and object 
 files at runtime) that is gc and exception friendly.  

I never understood why this is needed. Can't they simply compile the plugins into the main program? When it's a commercial program, the DLL plugin approach probably wouldn't work anyway: in order to enable others to compile plugins, you would need to expose your internal "headers" (D modules). Note that unlike in languages like C/C++, this would cause internal modules to be exposed too, even if they are not strictly needed. What would you do to avoid this? Maintain a separate set of import modules? I think a purely extern(C) based interface would be better in these cases. In fact, if you rely on the D ABI for dynamic linking, you'll probably have the same trouble as with C++ dynamic linking. For example, BeOS had to go through this to make sure their C++ based API maintains ABI compatibility: http://homepage.corbina.net/~maloff/holy-wars/fbc.html I'm not sure if the D ABI improves the situation. At any rate, it doesn't sound like a good idea.
Jan 28 2009
next sibling parent reply =?UTF-8?B?QWxleGFuZGVyIFDDoW5law==?= writes:
grauzone wrote:
 John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where multiple 
 applications have access to a single dll at runtime.  It appears that 
 such support would be quite difficult to implement and moves in the 
 direction of operating system features.

Couldn't this be achieved by simply mmap()-ing the file contents into memory? mmap() normally shared the memory pages with other processes. Of course, this wouldn't work if the code both isn't position independent, and needs to be relocated to a different base address. But that's also the case with operating system supported dynamic shared objects.
 It does do runtime linking, however, which is extremely useful for 
 certain situations... specifically any sort of application that needs 
 a plugin architecture for D (ie.. it can link with libraries and 
 object files at runtime) that is gc and exception friendly.  

I never understood why this is needed. Can't they simply compile the plugins into the main program?

Well, compiling them directly into the main program kinda defeats the purpose of runtime-pluggable plugins, wouldn’t it?
 When it's a commercial program, the DLL plugin approach probably 
 wouldn't work anyway: in order to enable others to compile plugins, you 
 would need to expose your internal "headers" (D modules). Note that 
 unlike in languages like C/C++, this would cause internal modules to be 
 exposed too, even if they are not strictly needed. What would you do to 
 avoid this? Maintain a separate set of import modules?

Make use of .di files. You don’t have to distribute code.
Jan 28 2009
parent reply grauzone <none example.net> writes:
Alexander Pánek wrote:
 grauzone wrote:
 John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where 
 multiple applications have access to a single dll at runtime.  It 
 appears that such support would be quite difficult to implement and 
 moves in the direction of operating system features.

Couldn't this be achieved by simply mmap()-ing the file contents into memory? mmap() normally shared the memory pages with other processes. Of course, this wouldn't work if the code both isn't position independent, and needs to be relocated to a different base address. But that's also the case with operating system supported dynamic shared objects.
 It does do runtime linking, however, which is extremely useful for 
 certain situations... specifically any sort of application that needs 
 a plugin architecture for D (ie.. it can link with libraries and 
 object files at runtime) that is gc and exception friendly.  

I never understood why this is needed. Can't they simply compile the plugins into the main program?

Well, compiling them directly into the main program kinda defeats the purpose of runtime-pluggable plugins, wouldn’t it?

But why do you want them in the first place? For Open Source projects, this seems to be completely pointless to me.
 When it's a commercial program, the DLL plugin approach probably 
 wouldn't work anyway: in order to enable others to compile plugins, 
 you would need to expose your internal "headers" (D modules). Note 
 that unlike in languages like C/C++, this would cause internal modules 
 to be exposed too, even if they are not strictly needed. What would 
 you do to avoid this? Maintain a separate set of import modules?

Make use of .di files. You don’t have to distribute code.

D interface files (.di) files are what I meant by "import module", sorry about this. They are compiler specific, and the only intended purpose is speeding up the compilation. Quoting from the D site:
 D interface files bear some analogous similarities to C++ header
 files. But they are not required in the way that C++ header files are,
 and they are not part of the D language. They are a feature of the
 compiler, and serve only as an optimization of the build process.

http://www.digitalmars.com/d/2.0/dmd-linux.html#interface_files I looked at the Tango .di files, which are (I think) automatically generated by the D compiler. I noticed several things: - Private symbols are included, even private imports or private class members => they are exposed to the public, and changing them might break ABI compatibility under circumstances. - All transitive imports seem to be included => you either expose your internal modules as interface file, or your public modules must not (transitively) import private modules. Note that this forbids direct use of any private type or function. You'll probably have to program around using indirections like interfaces or abstract base classes. Also note that this is way harder as in C: unlike in D, there are incomplete types, and the implementation (.c files) is completely separate and can import any private headers. - Sometimes, the full code for methods or functions is included, although they are not templated in any way. I guess it's even possible that plugins will inline those functions. This means changing these functions could randomly break already compiled plugins. Of course, this can be fixed, but nobody has bothered yet. It probably shows that .di files weren't designed with ABI compatibility in mind. It seems that dynamic linking with D code is extremely fragile, and it requires serious extra effort to maintain ABI compatibility. Please correct me if I'm wrong.
Jan 28 2009
next sibling parent Don <nospam nospam.com> writes:
grauzone wrote:
 Alexander Pánek wrote:
 grauzone wrote:
 John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where 
 multiple applications have access to a single dll at runtime.  It 
 appears that such support would be quite difficult to implement and 
 moves in the direction of operating system features.

Couldn't this be achieved by simply mmap()-ing the file contents into memory? mmap() normally shared the memory pages with other processes. Of course, this wouldn't work if the code both isn't position independent, and needs to be relocated to a different base address. But that's also the case with operating system supported dynamic shared objects.
 It does do runtime linking, however, which is extremely useful for 
 certain situations... specifically any sort of application that 
 needs a plugin architecture for D (ie.. it can link with libraries 
 and object files at runtime) that is gc and exception friendly.  

I never understood why this is needed. Can't they simply compile the plugins into the main program?

Well, compiling them directly into the main program kinda defeats the purpose of runtime-pluggable plugins, wouldn’t it?

But why do you want them in the first place? For Open Source projects, this seems to be completely pointless to me.
 When it's a commercial program, the DLL plugin approach probably 
 wouldn't work anyway: in order to enable others to compile plugins, 
 you would need to expose your internal "headers" (D modules). Note 
 that unlike in languages like C/C++, this would cause internal 
 modules to be exposed too, even if they are not strictly needed. What 
 would you do to avoid this? Maintain a separate set of import modules?

Make use of .di files. You don’t have to distribute code.

D interface files (.di) files are what I meant by "import module", sorry about this. They are compiler specific, and the only intended purpose is speeding up the compilation. Quoting from the D site: > D interface files bear some analogous similarities to C++ header > files. But they are not required in the way that C++ header files are, > and they are not part of the D language. They are a feature of the > compiler, and serve only as an optimization of the build process. http://www.digitalmars.com/d/2.0/dmd-linux.html#interface_files

They were also intended to act as header files, to conceal implementation details.
 I looked at the Tango .di files, which are (I think) automatically 
 generated by the D compiler. I noticed several things:
 
 - Private symbols are included, even private imports or private class 
 members => they are exposed to the public, and changing them might break 
 ABI compatibility under circumstances.
 
 - All transitive imports seem to be included => you either expose your 
 internal modules as interface file, or your public modules must not 
 (transitively) import private modules. Note that this forbids direct use
 of any private type or function. You'll probably have to program around 
 using indirections like interfaces or abstract base classes. Also note 
 that this is way harder as in C: unlike in D, there are incomplete 
 types, and the implementation (.c files) is completely separate and can 
 import any private headers.
 
 - Sometimes, the full code for methods or functions is included, 
 although they are not templated in any way. I guess it's even possible 
 that plugins will inline those functions.

Yes, I think that's why they're included. This means changing these
 functions could randomly break already compiled plugins. Of course, this 
 can be fixed, but nobody has bothered yet. It probably shows that .di 
 files weren't designed with ABI compatibility in mind.

 
 It seems that dynamic linking with D code is extremely fragile, and it 
 requires serious extra effort to maintain ABI compatibility. Please 
 correct me if I'm wrong.

Jan 28 2009
prev sibling parent "Nick Sabalausky" <a a.a> writes:
"grauzone" <none example.net> wrote in message 
news:glpha0$dg3$1 digitalmars.com...
 Alexander Pnek wrote:
 grauzone wrote:
 John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where multiple 
 applications have access to a single dll at runtime.  It appears that 
 such support would be quite difficult to implement and moves in the 
 direction of operating system features.

Couldn't this be achieved by simply mmap()-ing the file contents into memory? mmap() normally shared the memory pages with other processes. Of course, this wouldn't work if the code both isn't position independent, and needs to be relocated to a different base address. But that's also the case with operating system supported dynamic shared objects.
 It does do runtime linking, however, which is extremely useful for 
 certain situations... specifically any sort of application that needs a 
 plugin architecture for D (ie.. it can link with libraries and object 
 files at runtime) that is gc and exception friendly.

I never understood why this is needed. Can't they simply compile the plugins into the main program?

Well, compiling them directly into the main program kinda defeats the purpose of runtime-pluggable plugins, wouldn't it?

But why do you want them in the first place? For Open Source projects, this seems to be completely pointless to me.

With runtime-pluggables: 1. A program can download, install and use a plugin without the program needing to be restarted. 2. Out of all existing plugins, any arbitrary subset can be enabled without requiring an exponential number of pre-compiled executables, without requiring the user (or the program itself) to recompile anything, and without needing to install any build tools on the user's system.
Jan 28 2009
prev sibling next sibling parent reply Mike Parker <aldacron gmail.com> writes:
grauzone wrote:

 
 When it's a commercial program, the DLL plugin approach probably 
 wouldn't work anyway: in order to enable others to compile plugins, you 
 would need to expose your internal "headers" (D modules). Note that 

This is exactly what id software did with their Quake games. They provided an SDK, which included the headers and source files required to make mods. It was possible to use plain C to make DLLs or the QuakeC language they developed. This was before scripting languages became big in the game industry. Personally, I would prefer a scripting language like Lua or Python over DLLs/object files for a plugin framework, no matter the application domain. The biggest reason is that it's easier to sandbox a script engine. But both approaches have their place.
 unlike in languages like C/C++, this would cause internal modules to be 
 exposed too, even if they are not strictly needed. What would you do to 
 avoid this? Maintain a separate set of import modules?
 
 I think a purely extern(C) based interface would be better in these cases.
 
 In fact, if you rely on the D ABI for dynamic linking, you'll probably 
 have the same trouble as with C++ dynamic linking. For example, BeOS had 
 to go through this to make sure their C++ based API maintains ABI 
 compatibility:
 
 http://homepage.corbina.net/~maloff/holy-wars/fbc.html
 
 I'm not sure if the D ABI improves the situation. At any rate, it 
 doesn't sound like a good idea.

Jan 28 2009
next sibling parent Mike Parker <aldacron gmail.com> writes:
Mike Parker wrote:
 grauzone wrote:
 
 When it's a commercial program, the DLL plugin approach probably 
 wouldn't work anyway: in order to enable others to compile plugins, 
 you would need to expose your internal "headers" (D modules). Note that 

This is exactly what id software did with their Quake games. They provided an SDK, which included the headers and source files required to make mods. It was possible to use plain C to make DLLs or the QuakeC language they developed. This was before scripting languages became big in the game industry. Personally, I would prefer a scripting language like Lua or Python over DLLs/object files for a plugin framework, no matter the application domain. The biggest reason is that it's easier to sandbox a script engine. But both approaches have their place.

And I should qualify that I think the DLL/object file approach is the way to go for compiler plugins.
 
 
 unlike in languages like C/C++, this would cause internal modules to 
 be exposed too, even if they are not strictly needed. What would you 
 do to avoid this? Maintain a separate set of import modules?

 I think a purely extern(C) based interface would be better in these 
 cases.

 In fact, if you rely on the D ABI for dynamic linking, you'll probably 
 have the same trouble as with C++ dynamic linking. For example, BeOS 
 had to go through this to make sure their C++ based API maintains ABI 
 compatibility:

 http://homepage.corbina.net/~maloff/holy-wars/fbc.html

 I'm not sure if the D ABI improves the situation. At any rate, it 
 doesn't sound like a good idea.


Jan 28 2009
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Mike Parker" <aldacron gmail.com> wrote in message 
news:glpfr2$b5o$1 digitalmars.com...
 grauzone wrote:

 When it's a commercial program, the DLL plugin approach probably wouldn't 
 work anyway: in order to enable others to compile plugins, you would need 
 to expose your internal "headers" (D modules). Note that

This is exactly what id software did with their Quake games. They provided an SDK, which included the headers and source files required to make mods. It was possible to use plain C to make DLLs or the QuakeC language they developed. This was before scripting languages became big in the game industry. Personally, I would prefer a scripting language like Lua or Python over DLLs/object files for a plugin framework, no matter the application domain. The biggest reason is that it's easier to sandbox a script engine. But both approaches have their place.

I disagree. Making all add-ons be interpreted scripts is one of the biggest reasons why Firefox (especially v2) is so absurdly slow (not that I'm a fan of IE, Opera or Safari). Also, the fact that the vast majority of scripting languages lack descent compile-time checking (such as static type checking or mandatory explicit declarations), or at least push it off as a secondary concern (modern ECMAScript), creates a situation where plugins have a tendancy to be unreliable. But you're right that sandboxing is a potential issue. (Although even a scripting engine can still potentially contain exploits.)
Jan 28 2009
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Nick Sabalausky wrote:
 [snip]
 
 I disagree. Making all add-ons be interpreted scripts is one of the biggest 
 reasons why Firefox (especially v2) is so absurdly slow (not that I'm a fan 
 of IE, Opera or Safari). Also, the fact that the vast majority of scripting 
 languages lack descent compile-time checking (such as static type checking 
 or mandatory explicit declarations), or at least push it off as a secondary 
 concern (modern ECMAScript), creates a situation where plugins have a 
 tendancy to be unreliable.

There's an interesting talk Steve Yeggie gave a while back: http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html There's also a video of it: http://au.youtube.com/watch?v=tz-Bb-D6teE -- Daniel
Jan 28 2009
parent reply "Nick Sabalausky" <a a.a> writes:
"Daniel Keep" <daniel.keep.lists gmail.com> wrote in message 
news:glqsa7$13e2$1 digitalmars.com...
 Nick Sabalausky wrote:
 [snip]

 I disagree. Making all add-ons be interpreted scripts is one of the 
 biggest
 reasons why Firefox (especially v2) is so absurdly slow (not that I'm a 
 fan
 of IE, Opera or Safari). Also, the fact that the vast majority of 
 scripting
 languages lack descent compile-time checking (such as static type 
 checking
 or mandatory explicit declarations), or at least push it off as a 
 secondary
 concern (modern ECMAScript), creates a situation where plugins have a
 tendancy to be unreliable.

There's an interesting talk Steve Yeggie gave a while back: http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html There's also a video of it: http://au.youtube.com/watch?v=tz-Bb-D6teE -- Daniel

Hmm, yea, interesting, although his performance arguments are all about "potential" that, as he points out, isn't going to be realized any time soon for most scripting languages, and his compile-time-checking arguments seem...broken.
Jan 28 2009
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Nick Sabalausky wrote:
 "Daniel Keep" <daniel.keep.lists gmail.com> wrote in message 
 news:glqsa7$13e2$1 digitalmars.com...
 Nick Sabalausky wrote:
 [snip]

 I disagree. Making all add-ons be interpreted scripts is one of the 
 biggest
 reasons why Firefox (especially v2) is so absurdly slow (not that I'm a 
 fan
 of IE, Opera or Safari). Also, the fact that the vast majority of 
 scripting
 languages lack descent compile-time checking (such as static type 
 checking
 or mandatory explicit declarations), or at least push it off as a 
 secondary
 concern (modern ECMAScript), creates a situation where plugins have a
 tendancy to be unreliable.

http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html There's also a video of it: http://au.youtube.com/watch?v=tz-Bb-D6teE -- Daniel

Hmm, yea, interesting, although his performance arguments are all about "potential" that, as he points out, isn't going to be realized any time soon for most scripting languages, and his compile-time-checking arguments seem...broken.

(Warning: personal rambling ahead.) Well, the performance is an interesting one. There's a guy who wrote LuaJIT (a version of Lua that uses JIT compilation.) He's currently working on LuaJIT 2.0 that uses trace trees (like Yeggie mentioned.) Last time I checked, he'd had to change his benchmarking scheme to eschew scripting languages and benchmark directly against C. I think scripting languages have a lot of headroom left, that's just waiting for better interpreters and saner language design (I doubt you'll see JS or Python ever reach Lua's speeds, at least not any time soon.) As for Firefox, it's worth pointing out that the current release of LuaJIT is between 0 and ~12 times faster than TraceMonkey. Just because JavaScript is a crippled old dog doesn't mean you should write off all dynamic languages. :D Personally, I think the best approach is to combine the two; write the hot spots in C, D or some other fast language, and all the glue in a dynamic language. For all it's expressiveness, there are just some things that are easier to do with a dynamic language like Python or Lua than with D. And as for the compile-time checking thing... I think it's a bit of a wash. There are times that static checking is helpful... and times when it's too restrictive. All I know is that it's so much easier to debug something when you can get an interactive interpreter inside the debugger... -- Daniel
Jan 29 2009
next sibling parent reply grauzone <none example.net> writes:
Daniel Keep wrote:
 Personally, I think the best approach is to combine the two; write the
 hot spots in C, D or some other fast language, and all the glue in a
 dynamic language.  For all it's expressiveness, there are just some
 things that are easier to do with a dynamic language like Python or Lua
 than with D.

I agree, but it's hard to link all these languages together. They use different types, require bindings to import functions and classes, and all that. I heard writing Python modules in C is really hard.
 And as for the compile-time checking thing... I think it's a bit of a
 wash.  There are times that static checking is helpful... and times when
 it's too restrictive.  All I know is that it's so much easier to debug
 something when you can get an interactive interpreter inside the debugger...

Most debuggers seem to support a subset of C for evaluating expressions or even calling methods. For example, in gdb, you can write the command "print x->y->z". I don't think there's any substantial difference between dynamic and static languages here. It's all a question of effort, and static languages are just a bit behind. For example, if you had a compiler as library (wow, that's actually the subject of this thread), it wouldn't be so hard anymore to implement a read-eval-print-loop for a static language. Actually, Scala is a statically typed language, that provides such an interactive interpreter.
Jan 29 2009
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
grauzone wrote:
 Daniel Keep wrote:
 Personally, I think the best approach is to combine the two; write the
 hot spots in C, D or some other fast language, and all the glue in a
 dynamic language.  For all it's expressiveness, there are just some
 things that are easier to do with a dynamic language like Python or Lua
 than with D.

I agree, but it's hard to link all these languages together. They use different types, require bindings to import functions and classes, and all that. I heard writing Python modules in C is really hard.

Not in my experience. Pyrex makes it embarrassingly easy. I had a GUI app written in Python; at one point, I decided to improve performance and so I did a bit of profiling. The big hot spot was in the rendering code, so I re-wrote that part in C and used Pyrex to make the bridge. Compiled to a native Python extension module, and replaced the old pure Python version with it. Restart app and boom, it runs faster without having to touch a line of code anywhere else. Also, don't forget that Python has ctypes, which lets it dynamically bind to C libraries without having to actually write a wrapper. Now, going the OTHER way is a nightmare. [1]
 Most debuggers seem to support a subset of C for evaluating expressions
 or even calling methods. For example, in gdb, you can write the command
 "print x->y->z".
 
 I don't think there's any substantial difference between dynamic and
 static languages here. It's all a question of effort, and static
 languages are just a bit behind. For example, if you had a compiler as
 library (wow, that's actually the subject of this thread), it wouldn't
 be so hard anymore to implement a read-eval-print-loop for a static
 language. Actually, Scala is a statically typed language, that provides
 such an interactive interpreter.

Perhaps, but I'm yet to see any statically typed language that let me replace objects or even whole functions while the program's running. I'm not saying it can't be done, just that it's a lot harder for a statically typed and compiled language than one with dynamic lookup. -- Daniel [1] Actually, back when I was still playing around with Python bindings for D, one thing I started hacking on was a module for runtime codegen. The idea was to fire up the Python interpreter, poke around inside a Python module to find out what it might look like as a native function, then generate a stub that went from native code to Python code and back again. ... and then make it a DDL module so that you could treat Python libraries as if they were native code.
Jan 29 2009
parent bearophile <bearophileHUGS lycos.com> writes:
Daniel Keep:
 Not in my experience.  Pyrex makes it embarrassingly easy.

Today you use Cython, that is an improved variant of Pyrex. You can also use Pyd with D, that I have seen is quite easy to use. IHMO it deserves to be more widely known and used. Bye, bearophile
Jan 29 2009
prev sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:
 Personally, I think the best approach is to combine the two; write the
 hot spots in C, D or some other fast language, and all the glue in a
 dynamic language.  For all it's expressiveness, there are just some
 things that are easier to do with a dynamic language like Python or Lua
 than with D.

I keep hearing this stated, but I don't see very many use cases put forth where it is clearly better to use a dynamic language. For me, I'm not familiar with any dynamic languages to a sufficient degree to ever get an advantage from using them, but if I see a sufficient use case, I'll change that.
Jan 29 2009
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Bill Baxter (wbaxter gmail.com)'s article
 One thing, though, that I noticed using Python, is that a lot of what
 is considered so easy-to-use about dynamic languages is just that
 people are more willing to use expensive (but elegant) abstractions.
 If you're willing to do things in a slightly less efficient way, you
 can make the code look pretty darn elegant.  I think this is the basis
 of much of Bearophile's libs.  Most of the time you really don't need
 that extra performance.  With a dynamic language there's no way to
 really get the performance, anyway, so you might as well use something
 elegant.   People seem to have a harder time throwing away their
 desire for efficiency when they know their language *is* capable of
 being efficient.  I'm certainly included in that group.   I would have
 a hard time convincing myself to code in D similar to how I code in
 Python.  It just feels so sloppy to me when it's in D, because I know
 D can do better.   But in truth, most of the code I write would
 probably perform fine even with large helpings of extra slop.
 --bb

I feel the exact same way, but I thought I was just crazy. I don't use Python on a regular basis because most of the code I write has at least enough parts that have to be fast that it makes more sense to just write the whole thing in D. When I did use it a few times out of curiosity, I was like wow, some of this stuff is so much more elegant than the way I would do this in D. Then I thought for a little while longer and was like, "wait a minute, I _could_ do it that way in D, it's just that I never would because when I'm in D coding mode I'm so used to thinking about efficiency." Now, when I'm coding a part of a D program where performance isn't going to matter, like a function that gets run once at startup, I actively try to force myself _not_ to think about efficiency and to just keep it simple and elegant.
Jan 29 2009
parent bearophile <bearophileHUGS lycos.com> writes:
Bill Baxter:

One thing, though, that I noticed using Python, is that a lot of what is
considered so easy-to-use about dynamic languages is just that people are more
willing to use expensive (but elegant) abstractions. If you're willing to do
things in a slightly less efficient way, you can make the code look pretty darn
elegant.<

It's also a matter of handy syntax too, and having some type inferencing (or dynamic typing. Type inferencing may require a more complex type system, and a more complex and often slower compiler). I want to add two notes: 1) My work on ShedSkin (a partial Python => C++ compiler) has shown me that you can have a very handy syntax in statically typed C++-like language too. For example in Python you have list comps and generator expressions that are handy syntax, you need more code to write them in D: foo = [[x*y for x in someiterable] for y in somestring.split()] foo = ([x*y for x in someiterable] for y in somestring.split()) Other handy syntax is from iterators: def bar(): for x in xrange(10): yield x*x I miss those three things in D still. In my libs there are ways to do similar things, but they become quite more hairy and full of {}(), so in the end you may want to not use them at all in a real program. I hope D developers will eventually add list comps, iterators, and some other syntactic sugar to D. 2) Haskell compilers (and the Stalin Scheme compiler) show that if build a complex enough compiler, it can often be able to compile such abstractions to fast enough code. Not as fast as C code, but good enough for every noncritical spots in your program.
People seem to have a harder time throwing away their desire for efficiency
when they know their language *is* capable of being efficient.<

This requires you to know other languages, and to perform some self-control. Self-control is often one of the main things hat tell apart an adult professional person from a young newbie, outside the field of programming too. I think it can use to perform some "katas", shape exercises where you write the same code in various ways (and maybe you also benchmark them). ---------------------- dsimcha:
"wait a minute, I _could_ do it that way in D, it's just that I never would
because when I'm in D coding mode I'm so used to thinking about efficiency."<

That's why it's good to learn to program in different languages (Prolog, Lisp or Scheme, Haskell, Python, Erlang, C, C++, Mathematica, Java or C#, Oz or Mozart, a data flow language, and little else). You may need a life to learn them all :-) And when you know them, you may be too much old to do something good, that's why life is evil :-)
Now, when I'm coding a part of a D program where performance isn't going to
matter, like a function that gets run once at startup, I actively try to force
myself _not_ to think about efficiency and to just keep it simple and elegant.<

Because:
We should forget about small efficiencies, say about 97% of the time: premature
optimization is the root of all evil.<

Bye, bearophile
Jan 30 2009
prev sibling next sibling parent reply Yigal Chripun <yigal100 gmail.com> writes:
Bill Baxter wrote:
 On Fri, Jan 30, 2009 at 8:40 AM, Christopher Wright<dhasenan gmail.com>  wrote:
 Daniel Keep wrote:
 Personally, I think the best approach is to combine the two; write the
 hot spots in C, D or some other fast language, and all the glue in a
 dynamic language.  For all it's expressiveness, there are just some
 things that are easier to do with a dynamic language like Python or Lua
 than with D.

where it is clearly better to use a dynamic language. For me, I'm not familiar with any dynamic languages to a sufficient degree to ever get an advantage from using them, but if I see a sufficient use case, I'll change that.

Since Variants are kind of the basic variable type in dynamic languages, basically anywhere you can benefit from variants is a win for dynamic langs. More generally there are times when you can't really know what types things are going to be till run time. For instance dealing with a database. In a dynamic lang you can create new aggregate types on the fly that are just like built-in types. Or say you have an algorithm that can work on lots of different kinds of data types. But you need to load the data from a file at runtime, including the type. To do that with C++/D templates you have to pre-instantiate every possible type you want to support, even though most runs of your program will use only one or two instantiations of it. With a dynamic language you don't have to pre-instantiate anything, so such things are a lot easier. In GUIs when you need to loosely couple different components, dynamic langs can make life a lot easier. With std.signals in D, for instance, you have to be so anal about the call signatures. And you may have trouble calling a delegate that takes a bool when the signal sends out an int, etc Such things are rarely any problem for a dynamic language. Variable numbers of args are also generally very easy to use. Basically imagine any annoyance you have with D or C++. With a dynamic language it pretty much disappears. :-) (But of course you will encounter new issues, like performance, or higher testing burden because no types are statically checked). One thing, though, that I noticed using Python, is that a lot of what is considered so easy-to-use about dynamic languages is just that people are more willing to use expensive (but elegant) abstractions. If you're willing to do things in a slightly less efficient way, you can make the code look pretty darn elegant. I think this is the basis of much of Bearophile's libs. Most of the time you really don't need that extra performance. With a dynamic language there's no way to really get the performance, anyway, so you might as well use something elegant. People seem to have a harder time throwing away their desire for efficiency when they know their language *is* capable of being efficient. I'm certainly included in that group. I would have a hard time convincing myself to code in D similar to how I code in Python. It just feels so sloppy to me when it's in D, because I know D can do better. But in truth, most of the code I write would probably perform fine even with large helpings of extra slop. --bb

Static languages can have Variant/box types that'll give most of the same functionality of dynamic languages. so, instead of instantiating list!(int), list!(string), etc, you can get one list!(Variant).. C# provide such a feature with a keyword instead of rellying on templates. The real difference is that static languages have mostly read-only RTTI. (Java provides a very limited capability to reload classes, IIRC) a scripting language allows you to manipulate Types and instances at run-time, for example you can add/remove/alter methods in a class and affect all instances of that class, or alter a specific instance of a class. This cannot be done in a static language.
Jan 29 2009
next sibling parent reply grauzone <none example.net> writes:
Yigal Chripun wrote:
 Static languages can have Variant/box types that'll give most of the 
 same functionality of dynamic languages. so, instead of instantiating 
 list!(int), list!(string), etc, you can get one list!(Variant)..
 C# provide such a feature with a keyword instead of rellying on templates.

It would be nice if D would be extended to provide all these things. I guess the only reason this hasn't been done is the space overhead of full RTTI information. (And dynamic method invocation might require a lot of hackery to manually copy the method arguments on the stack.)
 The real difference is that static languages have mostly read-only RTTI. 
 (Java provides a very limited capability to reload classes, IIRC)
 a scripting language allows you to manipulate Types and instances at 
 run-time, for example you can add/remove/alter methods in a class and 
 affect all instances of that class, or alter a specific instance of a 
 class. This cannot be done in a static language.

Some static languages allow it to statically add your own methods or variables to a foreign class. An example would be AspectJ, which, among other things, adds this feature to Java. This is still not as dynamic as in dynamic type systems, but I wonder if you really need more? There's something similar in current static languages: you can extend the global data segment with global variables. Global variables don't have to be all in a single source file. Instead, the linker takes care of collecting global variables from object files and allocating space for them in the data segment. A dynamic linker can do this even while a program is running! Thread local storage (with global __thread variables) does the same thing per thread. Threads can be created and destroyed at runtime. Dynamic linking is still supported, I think.
Jan 30 2009
parent reply Yigal Chripun <yigal100 gmail.com> writes:
grauzone wrote:
 Yigal Chripun wrote:
 Static languages can have Variant/box types that'll give most of the
 same functionality of dynamic languages. so, instead of instantiating
 list!(int), list!(string), etc, you can get one list!(Variant)..
 C# provide such a feature with a keyword instead of rellying on
 templates.

It would be nice if D would be extended to provide all these things. I guess the only reason this hasn't been done is the space overhead of full RTTI information. (And dynamic method invocation might require a lot of hackery to manually copy the method arguments on the stack.)
 The real difference is that static languages have mostly read-only
 RTTI. (Java provides a very limited capability to reload classes, IIRC)
 a scripting language allows you to manipulate Types and instances at
 run-time, for example you can add/remove/alter methods in a class and
 affect all instances of that class, or alter a specific instance of a
 class. This cannot be done in a static language.

Some static languages allow it to statically add your own methods or variables to a foreign class. An example would be AspectJ, which, among other things, adds this feature to Java.

from what little I know about AOP - it's done at link time, not runtime. Java does however provide very limited support for this - for example, while debugging your code, you can change an implementation of a method and reload the containing class so instances of that class *that where created after the change* will call the new implementation. this allows to make small changes while debugging without the need to re-run the program each time. But you can't do things like: Class a = new Class; class b = new Class; a.getClass.addMethod(foo); b.foo(); note that this is done at run-time, not link time.
 This is still not as dynamic as in dynamic type systems, but I wonder if
 you really need more?

 There's something similar in current static languages: you can extend
 the global data segment with global variables. Global variables don't
 have to be all in a single source file. Instead, the linker takes care
 of collecting global variables from object files and allocating space
 for them in the data segment. A dynamic linker can do this even while a
 program is running!

 Thread local storage (with global __thread variables) does the same
 thing per thread. Threads can be created and destroyed at runtime.
 Dynamic linking is still supported, I think.

Can you explain more? I don't see how globals and threads can allow me to do the above code snippet.
Jan 31 2009
parent grauzone <none example.net> writes:
Yigal Chripun wrote:
 grauzone wrote:
 Yigal Chripun wrote:
 Static languages can have Variant/box types that'll give most of the
 same functionality of dynamic languages. so, instead of instantiating
 list!(int), list!(string), etc, you can get one list!(Variant)..
 C# provide such a feature with a keyword instead of rellying on
 templates.

It would be nice if D would be extended to provide all these things. I guess the only reason this hasn't been done is the space overhead of full RTTI information. (And dynamic method invocation might require a lot of hackery to manually copy the method arguments on the stack.)
 The real difference is that static languages have mostly read-only
 RTTI. (Java provides a very limited capability to reload classes, IIRC)
 a scripting language allows you to manipulate Types and instances at
 run-time, for example you can add/remove/alter methods in a class and
 affect all instances of that class, or alter a specific instance of a
 class. This cannot be done in a static language.

Some static languages allow it to statically add your own methods or variables to a foreign class. An example would be AspectJ, which, among other things, adds this feature to Java.

from what little I know about AOP - it's done at link time, not runtime. Java does however provide very limited support for this - for example, while debugging your code, you can change an implementation of a method and reload the containing class so instances of that class *that where created after the change* will call the new implementation. this allows to make small changes while debugging without the need to re-run the program each time. But you can't do things like: Class a = new Class; class b = new Class; a.getClass.addMethod(foo); b.foo(); note that this is done at run-time, not link time.

Yes. But when do you need to be fully dynamic? I claim that this is unneeded in most cases. Most time, you probably only want to extend a class by your own methods and members. To me, it looks like the biggest fundamental advantage of dynamic languages is only to be able to omit type signatures.
 This is still not as dynamic as in dynamic type systems, but I wonder if
 you really need more?

 There's something similar in current static languages: you can extend
 the global data segment with global variables. Global variables don't
 have to be all in a single source file. Instead, the linker takes care
 of collecting global variables from object files and allocating space
 for them in the data segment. A dynamic linker can do this even while a
 program is running!

 Thread local storage (with global __thread variables) does the same
 thing per thread. Threads can be created and destroyed at runtime.
 Dynamic linking is still supported, I think.

Can you explain more? I don't see how globals and threads can allow me to do the above code snippet.

First, this was only meant as some kind of analogy between dynamic languages and the AOP techniques mentioned above. Like you can dynamically add global variables to your program by loading a dynamic shared object (.so/.dll), although there aren't functions like addGlobalVariable(char[] name, size_t size). A thread is almost like an object, and declaring __thread variables is like adding fields to this object. Second, concerning actual implementations, you maybe could turn every class into a separate data section. Then it would only require a new linker symbol to extend a class by a field. The linker would take care of doing the actual class layout by assigning each linker symbol an address. This address is used as field offset. But I don't know enough about linkers to tell if this would actually work. And there are still some problems, like the question what happens with dynamic linking.
Jan 31 2009
prev sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Yigal Chripun (yigal100 gmail.com)'s article
 Static languages can have Variant/box types that'll give most of the
 same functionality of dynamic languages. so, instead of instantiating
 list!(int), list!(string), etc, you can get one list!(Variant)..
 C# provide such a feature with a keyword instead of rellying on templates.
 The real difference is that static languages have mostly read-only RTTI.
 (Java provides a very limited capability to reload classes, IIRC)
 a scripting language allows you to manipulate Types and instances at
 run-time, for example you can add/remove/alter methods in a class and
 affect all instances of that class, or alter a specific instance of a
 class. This cannot be done in a static language.

Out of curiosity, does anyone actually use Variant in D? When I was new to the language, I thought it was a great idea, but then I discovered D templates, so now I never use it.
Jan 30 2009
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
dsimcha wrote:
 == Quote from Yigal Chripun (yigal100 gmail.com)'s article
 Static languages can have Variant/box types that'll give most of the
 same functionality of dynamic languages. so, instead of instantiating
 list!(int), list!(string), etc, you can get one list!(Variant)..
 C# provide such a feature with a keyword instead of rellying on templates.
 The real difference is that static languages have mostly read-only RTTI.
 (Java provides a very limited capability to reload classes, IIRC)
 a scripting language allows you to manipulate Types and instances at
 run-time, for example you can add/remove/alter methods in a class and
 affect all instances of that class, or alter a specific instance of a
 class. This cannot be done in a static language.

Out of curiosity, does anyone actually use Variant in D? When I was new to the language, I thought it was a great idea, but then I discovered D templates, so now I never use it.

The only case where I've really used it* was, not coincidentally, the reason I wrote it in the first place: a generalised CVar system for a game engine. Really, you can pretty easily get away with never needing it. If you want runtime polymorphism and you're only storing class instances, then you can just use Object instead. Variant is really only useful if you want to store non-class types as well without having to Box them, or you really want value semantics. Still, it's cool that it works as well as it does... :D -- Daniel * I'm referring to Tango's Variant, not Phobos'.
Jan 30 2009
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Bill Baxter wrote:
 Does Tango's Variant have a fixed type?

No; it's like Phobos'.
 It seems the std2 Variant doesn't really care what the type of the
 thing you stuff in it is, as long as it fits in the memory space
 allotted.   How is that useful?   What's the use case for needing
 something that can be either 2.4f or "fred"?    (Sorry I don't know
 what a "CVar" system is...)

Think about Quake, or anything based on Id's engines. CVars are basically global variables you can set from the in-game console. The original problem was this: "I want a hash indexed by string going to... um... er... anything!" And thus, Variant was born. Incidentally, I've since changed to a design involving callbacks, so there you go. I suppose that it's, in a way, like D's support for typesafe variadic functions; except it only takes one value. :P
 What I actually needed was something
 with a fixed, internal type that could expose its value in a flexible
 way via templated get/set routines.  But for me a float property is
 never going to mutate into a string property.
 
 --bb

OK, your turn: why would you want something that wraps a single type in itself and has to be accessed via templates? Couldn't you just... use the type you want with templates? -- Daniel
Jan 30 2009
parent Daniel Keep <daniel.keep.lists gmail.com> writes:
Bill Baxter wrote:
 [snip]

This is really interesting, since you've almost described the solution I moved to after Variants (with rather ironically still uses Variants, but that's just an implementation detail now :P). The problem for me was that I wanted to be able to expose various internal bits of state to the console by string name. I also wanted to KNOW when something had been changed. The console had to be able to get, set and copy those values as both strings and their native types. Basically, random parts of the engine and game code could register a block of callback functions with the central cvar registry; the struct looks like this: struct ConfigVarCallbacks { Variant delegate() getValue; void delegate(Variant) setValue; Variant delegate(char[]) fromRepr; char[] delegate(Variant) toRepr; } The above was automatically generated by templates. Incidentally, THAT is was spurred me to write Tango's to!(T) template. :D It was used like this (mostly; I think I lost the implementation of cvars, but the templates and CTFE underlying it are still there): class Window { mixin(cvars( ` uint width : window.width = 800; uint height : window.height = 600; ` )); this() { _cvars_register; } } This would generate the storage for the cvar as a member of Window, generate the callback functions, and then do all the registration boilerplate. With that, you could talk to the config object to play with those values. { config.setVarString("window.width", "1024"); config.setVar("window.height", 768u); writefln("Window size: %sx%s", config.getVar("window.width").get!(uint), config.getVarString("window.height")); } And of course, the console can now get/set values from raw strings, or copy values between cvars in their native type via Variants. The one thing that this system never accounted for was user-created cvars which could potentially be... Variants. :P -- Daniel
Jan 30 2009
prev sibling parent grauzone <none example.net> writes:
dsimcha wrote:
 == Quote from Yigal Chripun (yigal100 gmail.com)'s article
 Static languages can have Variant/box types that'll give most of the
 same functionality of dynamic languages. so, instead of instantiating
 list!(int), list!(string), etc, you can get one list!(Variant)..
 C# provide such a feature with a keyword instead of rellying on templates.
 The real difference is that static languages have mostly read-only RTTI.
 (Java provides a very limited capability to reload classes, IIRC)
 a scripting language allows you to manipulate Types and instances at
 run-time, for example you can add/remove/alter methods in a class and
 affect all instances of that class, or alter a specific instance of a
 class. This cannot be done in a static language.

Out of curiosity, does anyone actually use Variant in D? When I was new to the language, I thought it was a great idea, but then I discovered D templates, so now I never use it.

I use a Box (Variant with less features) for some kind of command line parser. First, one can register a callback to handle a command. This callback takes a Box[] to carry the command arguments. When you register a command, you pass a TypeInfo[] to tell the parser how many argument there are and what type they must have (the Box[] types will be exactly the same as in TypeInfo[]). The catch is, that the parser can automatically display help messages and useful error messages if parsing fails. The command callback doesn't need to care about this. Second, you can register argument parsers. An argument parser is just a simple callback again. It takes a string and returns a Box. So you can add parsers for types, which are completely unknown to the command line parser code. This is nice and simple. How would you do it without Box?
Jan 31 2009
prev sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Sat, Jan 31, 2009 at 2:52 PM, Daniel Keep
<daniel.keep.lists gmail.com> wrote:
 Bill Baxter wrote:
 [snip]


Thanks for taking the time to explain your design. Some interesting differences there, like your exposing the variant directly instead of forwarding to it. I was trying to think how I could make my Property interface into an actual D interface when templates are part of the interface, and those can't be virtual. But the answer is just what you're doing: put the templates inside a struct -- expose the Variant in the interface and let *it* provide the templates.
 This is really interesting, since you've almost described the solution I
 moved to after Variants (with rather ironically still uses Variants, but
 that's just an implementation detail now :P).

Sounds like it's really the same problem, so maybe not so surprising the solution looks similar. You've got a console from which users can view and set state. I've got a properties panel. Basically the same thing. --bb
Jan 31 2009
prev sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Sat, Jan 31, 2009 at 11:48 AM, Daniel Keep
<daniel.keep.lists gmail.com> wrote:
 Bill Baxter wrote:
 Does Tango's Variant have a fixed type?

No; it's like Phobos'.
 It seems the std2 Variant doesn't really care what the type of the
 thing you stuff in it is, as long as it fits in the memory space
 allotted.   How is that useful?   What's the use case for needing
 something that can be either 2.4f or "fred"?    (Sorry I don't know
 what a "CVar" system is...)

Think about Quake, or anything based on Id's engines. CVars are basically global variables you can set from the in-game console. The original problem was this: "I want a hash indexed by string going to... um... er... anything!" And thus, Variant was born. Incidentally, I've since changed to a design involving callbacks, so there you go.

Ok I see.
 I suppose that it's, in a way, like D's support for typesafe variadic
 functions; except it only takes one value. :P

 What I actually needed was something
 with a fixed, internal type that could expose its value in a flexible
 way via templated get/set routines.  But for me a float property is
 never going to mutate into a string property.

 --bb

OK, your turn: why would you want something that wraps a single type in itself and has to be accessed via templates? Couldn't you just... use the type you want with templates?

I'm using it like a property in a GUI. The property is float or bool or whatever, I just need to be able to get and set the value in some generic way. So at least it's gotta have conversions to and from string -- to for showing in the GUI, from for turning the user's edit back into a value. And additionally it would be nice for it to not be too much a stickler for exact type -- to be able to convert to and from int even if its really a float, for example. I thought that was the kind of problem Variant was supposed to solve, but seems not quite so. But my original vision was that the base Property type would have template methods that pretty much like Variant's. get(T) opAssign(T). I don't see how to make that work without having a base class that contains the actual value in somehting like a Variant. So right now a sketch of the design is an abstract Property with concrete Properties with specific type derived from that. Like so: class Property { Variant value; T get(T) { value.get!(T)(); } void opAssign(T)(T v) { value = v; } abstract void fromString(string str); } class PropertyT(T) : Property { void fromString(string str) { ... } } So really it's pretty much like your CVar case. But the difference is a Property should only be one particular thing for its entire life, whereas the thing pointed to by "foo" in your hash map can change type. --bb
Jan 30 2009
prev sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Fri, Jan 30, 2009 at 11:35 PM, Daniel Keep
<daniel.keep.lists gmail.com> wrote:
 Out of curiosity, does anyone actually use Variant in D?  When I was new to the
 language, I thought it was a great idea, but then I discovered D templates, so
now
 I never use it.

The only case where I've really used it* was, not coincidentally, the reason I wrote it in the first place: a generalised CVar system for a game engine. Really, you can pretty easily get away with never needing it. If you want runtime polymorphism and you're only storing class instances, then you can just use Object instead. Variant is really only useful if you want to store non-class types as well without having to Box them, or you really want value semantics. Still, it's cool that it works as well as it does... :D -- Daniel * I'm referring to Tango's Variant, not Phobos'.

Does Tango's Variant have a fixed type? It seems the std2 Variant doesn't really care what the type of the thing you stuff in it is, as long as it fits in the memory space allotted. How is that useful? What's the use case for needing something that can be either 2.4f or "fred"? (Sorry I don't know what a "CVar" system is...) What I actually needed was something with a fixed, internal type that could expose its value in a flexible way via templated get/set routines. But for me a float property is never going to mutate into a string property. --bb
Jan 30 2009
prev sibling next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
grauzone wrote:
 John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where multiple
 applications have access to a single dll at runtime.  It appears that
 such support would be quite difficult to implement and moves in the
 direction of operating system features.

Couldn't this be achieved by simply mmap()-ing the file contents into memory? mmap() normally shared the memory pages with other processes.

I'm sure users of DDL would love for you to submit a patch. :)
 Of course, this wouldn't work if the code both isn't position
 independent, and needs to be relocated to a different base address. But
 that's also the case with operating system supported dynamic shared
 objects.
 
 It does do runtime linking, however, which is extremely useful for
 certain situations... specifically any sort of application that needs
 a plugin architecture for D (ie.. it can link with libraries and
 object files at runtime) that is gc and exception friendly.  

I never understood why this is needed. Can't they simply compile the plugins into the main program?

A plugin architecture, by definition, is to let third parties add code to your application. This rather precludes being able to simply link it in.
 When it's a commercial program, the DLL plugin approach probably
 wouldn't work anyway: in order to enable others to compile plugins, you
 would need to expose your internal "headers" (D modules). Note that
 unlike in languages like C/C++, this would cause internal modules to be
 exposed too, even if they are not strictly needed. What would you do to
 avoid this? Maintain a separate set of import modules?

As Alexander Pánek said; you can just use .di files, which the compiler can create from existing .d modules. Heck, you can just use regular .d modules and stub out the implementations. There's no reason why you'd need to release your code to third parties. Templates are a different matter, but then C++ has the same problem. Whether or not you want to release your templates as part of the SDK really depends on what they are. Templates + interfaces make a good pair.
 I think a purely extern(C) based interface would be better in these cases.
 
 In fact, if you rely on the D ABI for dynamic linking, you'll probably
 have the same trouble as with C++ dynamic linking. For example, BeOS had
 to go through this to make sure their C++ based API maintains ABI
 compatibility:
 
 http://homepage.corbina.net/~maloff/holy-wars/fbc.html
 
 I'm not sure if the D ABI improves the situation. At any rate, it
 doesn't sound like a good idea.

There's some interesting problems at that article that maybe we should ask Walter about. For one, being able to control virtual function ordering is an interesting idea. But by and large, these are the same problems you'll get with ANY ABI. If the size of a struct changes in a C ABI, you're hosed just as bad as if the size of a struct in D, or the fields of a class. -- Daniel
Jan 28 2009
parent reply grauzone <none example.net> writes:
Daniel Keep wrote:
 
 grauzone wrote:
 John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where multiple
 applications have access to a single dll at runtime.  It appears that
 such support would be quite difficult to implement and moves in the
 direction of operating system features.

memory? mmap() normally shared the memory pages with other processes.

I'm sure users of DDL would love for you to submit a patch. :)
 Of course, this wouldn't work if the code both isn't position
 independent, and needs to be relocated to a different base address. But
 that's also the case with operating system supported dynamic shared
 objects.

 It does do runtime linking, however, which is extremely useful for
 certain situations... specifically any sort of application that needs
 a plugin architecture for D (ie.. it can link with libraries and
 object files at runtime) that is gc and exception friendly.  

plugins into the main program?

A plugin architecture, by definition, is to let third parties add code to your application. This rather precludes being able to simply link it in.

But why go through the trouble of doing dynamic linking? You still can have a plugin architecture without dynamic linking.
 When it's a commercial program, the DLL plugin approach probably
 wouldn't work anyway: in order to enable others to compile plugins, you
 would need to expose your internal "headers" (D modules). Note that
 unlike in languages like C/C++, this would cause internal modules to be
 exposed too, even if they are not strictly needed. What would you do to
 avoid this? Maintain a separate set of import modules?

As Alexander Pánek said; you can just use .di files, which the compiler can create from existing .d modules.

See my reply to his posting.
 Heck, you can just use regular .d modules and stub out the
 implementations.  There's no reason why you'd need to release your code
 to third parties.

Sure, if you like tedious work. And you have to be extremely careful. Writing an extern(C) based interface will be not much more work, but the result will be much more robust, and give you some extra advantages, like enabling other languages to link to it.
 Templates are a different matter, but then C++ has the same problem.
 Whether or not you want to release your templates as part of the SDK
 really depends on what they are.  Templates + interfaces make a good pair.

Sure. Bytecode based languages like C# still can export compiled generics, though. Of course, they are not as flexible as D or C++ templates.
 I think a purely extern(C) based interface would be better in these cases.

 In fact, if you rely on the D ABI for dynamic linking, you'll probably
 have the same trouble as with C++ dynamic linking. For example, BeOS had
 to go through this to make sure their C++ based API maintains ABI
 compatibility:

 http://homepage.corbina.net/~maloff/holy-wars/fbc.html

 I'm not sure if the D ABI improves the situation. At any rate, it
 doesn't sound like a good idea.

There's some interesting problems at that article that maybe we should ask Walter about. For one, being able to control virtual function ordering is an interesting idea. But by and large, these are the same problems you'll get with ANY ABI. If the size of a struct changes in a C ABI, you're hosed just as bad as if the size of a struct in D, or the fields of a class.

I'm not sure, but I think you could solve many of these problems by using linker symbols instead of compile time constants for sizes and field offsets. Still doesn't work in D, because CTFE still needs them as real compile time constants. And it'd probably be less efficient.
   -- Daniel

Jan 28 2009
parent "Nick Sabalausky" <a a.a> writes:
"grauzone" <none example.net> wrote in message 
news:glphtn$g0f$1 digitalmars.com...
 Daniel Keep wrote:
 Templates are a different matter, but then C++ has the same problem.
 Whether or not you want to release your templates as part of the SDK
 really depends on what they are.  Templates + interfaces make a good 
 pair.

Sure. Bytecode based languages like C# still can export compiled generics, though. Of course, they are not as flexible as D or C++ templates.

AIUI, C#'s ability to export compiled generics forced the generics to be designed in such a way that objects of the generic type 'T' can only be used in ways explicitly allowed by T's explicit constraints. For instance, performing a comparison on an object of type T is a compile-time error unless the programmer places a constraint on T that T must be something that implements the IComparable interface (which includes all of the primitives). As far as I can tell, the only limitation this inherently forces on generics is that the language/library must provide a constraint for anything that a template is able to do. The main part where this *really* gimps C#'s generics is just simply MS's constant refusal to provide either an IArithmetic counterpart to IComparable or individual operator constraints. In other words, I'm not certain that the lack of flexibility in C#'s generics is an inherent consequence of its compiled generics being exportable.
Jan 28 2009
prev sibling parent Christopher Wright <dhasenan gmail.com> writes:
grauzone wrote:
 John Reimer wrote:
 ddl does not work for memory sharing like normal dll's, where multiple 
 applications have access to a single dll at runtime.  It appears that 
 such support would be quite difficult to implement and moves in the 
 direction of operating system features.

Couldn't this be achieved by simply mmap()-ing the file contents into memory? mmap() normally shared the memory pages with other processes.

DDL, after loading the library into RAM, has to fix up type information. This requires altering the loaded data. Therefore, DDL can't share everything, and the things it can share are fragmented.
Jan 28 2009
prev sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Fri, Jan 30, 2009 at 8:40 AM, Christopher Wright <dhasenan gmail.com> wrote:
 Daniel Keep wrote:
 Personally, I think the best approach is to combine the two; write the
 hot spots in C, D or some other fast language, and all the glue in a
 dynamic language.  For all it's expressiveness, there are just some
 things that are easier to do with a dynamic language like Python or Lua
 than with D.

I keep hearing this stated, but I don't see very many use cases put forth where it is clearly better to use a dynamic language. For me, I'm not familiar with any dynamic languages to a sufficient degree to ever get an advantage from using them, but if I see a sufficient use case, I'll change that.

Since Variants are kind of the basic variable type in dynamic languages, basically anywhere you can benefit from variants is a win for dynamic langs. More generally there are times when you can't really know what types things are going to be till run time. For instance dealing with a database. In a dynamic lang you can create new aggregate types on the fly that are just like built-in types. Or say you have an algorithm that can work on lots of different kinds of data types. But you need to load the data from a file at runtime, including the type. To do that with C++/D templates you have to pre-instantiate every possible type you want to support, even though most runs of your program will use only one or two instantiations of it. With a dynamic language you don't have to pre-instantiate anything, so such things are a lot easier. In GUIs when you need to loosely couple different components, dynamic langs can make life a lot easier. With std.signals in D, for instance, you have to be so anal about the call signatures. And you may have trouble calling a delegate that takes a bool when the signal sends out an int, etc Such things are rarely any problem for a dynamic language. Variable numbers of args are also generally very easy to use. Basically imagine any annoyance you have with D or C++. With a dynamic language it pretty much disappears. :-) (But of course you will encounter new issues, like performance, or higher testing burden because no types are statically checked). One thing, though, that I noticed using Python, is that a lot of what is considered so easy-to-use about dynamic languages is just that people are more willing to use expensive (but elegant) abstractions. If you're willing to do things in a slightly less efficient way, you can make the code look pretty darn elegant. I think this is the basis of much of Bearophile's libs. Most of the time you really don't need that extra performance. With a dynamic language there's no way to really get the performance, anyway, so you might as well use something elegant. People seem to have a harder time throwing away their desire for efficiency when they know their language *is* capable of being efficient. I'm certainly included in that group. I would have a hard time convincing myself to code in D similar to how I code in Python. It just feels so sloppy to me when it's in D, because I know D can do better. But in truth, most of the code I write would probably perform fine even with large helpings of extra slop. --bb
Jan 29 2009