www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Does a Interpretation-Engine fit in phobos ?

reply Stefan Koch <uplink.coder googlemail.com> writes:
Hi,

I recently had a breakthrough in my CTFE work.

Though because habits die hard.
I am writing the bytecode-engine in a CTFEable style.
Therefore I can be used as a quite comfortable IR for CTFE things 
as well.

It should be fairly easy to generate a inline-asm-string from the 
byteCode  at compiletime.
And thereby creating the possibility of generating optimized code 
at compile-time.

I was wondering if such a thing would fit in phobos.

Please share your opinions with me :)

Regards,
Stefan
Jun 30 2016
next sibling parent reply qznc <qznc web.de> writes:
On Thursday, 30 June 2016 at 10:28:03 UTC, Stefan Koch wrote:

 Hi,

 I recently had a breakthrough in my CTFE work.

 Though because habits die hard.
 I am writing the bytecode-engine in a CTFEable style.
 Therefore I can be used as a quite comfortable IR for CTFE 
 things as well.

 It should be fairly easy to generate a inline-asm-string from 
 the byteCode  at compiletime.
 And thereby creating the possibility of generating optimized 
 code at compile-time.

 I was wondering if such a thing would fit in phobos.

 Please share your opinions with me :)

 Regards,
 Stefan


Any sufficiently complicated C or Fortran [or D] program contains 
an ad hoc, informally-specified, bug-ridden, slow implementation 
of half of Common Lisp. –Greenspun's tenth rule

Ok, seriously, it sounds like an awesome feat, but I don't think 
it is necessary to put it into Phobos. First, a dub package, 
please.

Off-topic: Is it possible/feasible/desirable to let dmd use dub 
packages? I guess it would make sense to extract parts of dmd 
into dub packages. As a next step, dmd could use those packages 
instead of duplicating code.
Jun 30 2016
next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:

 Off-topic: Is it possible/feasible/desirable to let dmd use dub 
 packages?


please, no. not everybody out there is dub fan.
Jun 30 2016
next sibling parent Seb <seb wilzba.ch> writes:
On Thursday, 30 June 2016 at 11:09:10 UTC, ketmar wrote:

 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:

 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?


 please, no. not everybody out there is dub fan.


Off-topic:

 ketmar: if it's on dub (d's official package manager), it's 
easily visible to others and searchable.
Moreover most people prefer to have a dependency manager to 
download the sources and compile it. Can't you just grab the 
referenced source repository manually?

Please open a new thread if that doesn't work for you ;-)
Jun 30 2016
prev sibling parent reply lobo <swamplobo gmail.com> writes:
On Thursday, 30 June 2016 at 11:09:10 UTC, ketmar wrote:

 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:

 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?


 please, no. not everybody out there is dub fan.


You can always use dub to fetch packages into ~/.dub/packages and 
build them. The artifacts are then available for whatever build 
system you choose for your own project.

I don't use DUB for my own projects because it drops garbage all 
through my source tree. The day it can do out of source builds is 
the day I'll revisit the DUB bazaar.


bye,
lobo
Jun 30 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 30 June 2016 at 23:27:10 UTC, lobo wrote:

 On Thursday, 30 June 2016 at 11:09:10 UTC, ketmar wrote:

 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:

 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?


 please, no. not everybody out there is dub fan.


 You can always use dub to fetch packages into ~/.dub/packages 
 and build them. The artifacts are then available for whatever 
 build system you choose for your own project.

 I don't use DUB for my own projects because it drops garbage 
 all through my source tree. The day it can do out of source 
 builds is the day I'll revisit the DUB bazaar.


 bye,
 lobo


The Bytecode interpreter is now CTFEable :)
Jul 02 2016
parent reply Joakim =?UTF-8?B?QnLDpG5uc3Ryw7Zt?= <notfornow dev.null.com> writes:
On Saturday, 2 July 2016 at 08:11:48 UTC, Stefan Koch wrote:

 The Bytecode interpreter is now CTFEable :)


Good job Stefan. The PR looks really interesting.
I'll be looking into using your engine I future projects so I 
hope that it will be available in some way or another :)

// Joakim
Jul 02 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 2 July 2016 at 09:22:48 UTC, Joakim Brännström wrote:

 On Saturday, 2 July 2016 at 08:11:48 UTC, Stefan Koch wrote:

 The Bytecode interpreter is now CTFEable :)


 Good job Stefan. The PR looks really interesting.
 I'll be looking into using your engine I future projects so I 
 hope that it will be available in some way or another :)

 // Joakim


Good it hear.
I just pulled out a lot of the generation functions.
And made sure the "Disassembler" is ctfeable as well.
Jul 02 2016
prev sibling parent reply Wyatt <wyatt.epp gmail.com> writes:
On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:

 Ok, seriously, it sounds like an awesome feat, but I don't 
 think it is necessary to put it into Phobos. First, a dub 
 package, please.


Agree.  Does Java even have something like that?  That's sort of 
the exemplar for "hopelessly overdone standard library".


 Off-topic: Is it possible/feasible/desirable to let dmd use dub 
 packages?


DMD shouldn't have to download things from the public internet to 
do its job.


 I guess it would make sense to extract parts of dmd into dub 
 packages. As a next step, dmd could use those packages instead 
 of duplicating code.


Does it?  Which parts?  I'm afraid I don't see the benefit.

-Wyatt
Jun 30 2016
parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Thursday, 30 June 2016 at 14:14:41 UTC, Wyatt wrote:

 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:

 Ok, seriously, it sounds like an awesome feat, but I don't 
 think it is necessary to put it into Phobos. First, a dub 
 package, please.


 Agree.  Does Java even have something like that?


I'm afraid you couldn't be more wrong. Both Java and .NET have 
many provide ways for generating and executing byte code at 
run-time. Sometimes this the only way to implement something 
efficiently when runtime reflection is needed. See for example:
https://msdn.microsoft.com/en-us/library/mt654263.aspx,
https://msdn.microsoft.com/en-us/library/mt654267.aspx,
http://openjdk.java.net/groups/compiler/doc/package-overview/index.html,
http://asm.ow2.org/,
https://commons.apache.org/proper/commons-bcel/manual.html,
https://github.com/cglib/cglib,
http://jboss-javassist.github.io/javassist/,
https://www.jetbrains.com/mps/index.html,
https://docs.microsoft.com/en-us/dotnet/core/api/system.reflection.emit and
https://www.postsharp.net/features

Also almost every dynamic language has some sort of eval function 
can be used to evaluate arbitrary code at run-time. This has it's 
security and maintainability challenges, but without any doubt 
there are situations where this is very helpful.


 That's sort of the exemplar for "hopelessly overdone standard 
 library".


LOL, I have never heard about a user complaining that a product 
has too many features, as long as they don't get in the way. 
Instead users complain when something is **not available**, 
because that prevents them from getting their job done. What's 
wrong with having a module in the standard library that you 
personally won't use, but others will find helpful?

There are many ways in which code-generation can interact with 
the language runtime. By including it in the standard library, we 
can ensure that it is thoroughly tested on all supported 
platforms. Of course the other benefit is that it can be used 
from other modules in the standard library for implementing 
various optimizations (e.g. optimizing regex, linear algebra, 
data base queries, etc.) Such functionality has been a huge 
success for .NET. E.g. they enabled some advanced LINQ features 
which are used under the hood of almost every .NET project.


 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?


 DMD shouldn't have to download things from the public internet 
 to do its job.


I don't think you understood the question. The question is how 
should DMD's code base be structured / modularized. Of course 
after the DMD is compiled it shouldn't need to use the internet, 
but that's not the point. The question is if it's a good idea to 
split the project in small safe-contained reusable packages. For 
example, that would allow linters to leverage the compiler lexer 
and parser instead of implementing their own, which often can't 
handle all language features.

Another huge area is compiler plugins which are quite popular in 
Rust and .NET
https://doc.rust-lang.org/book/compiler-plugins.html
https://github.com/dotnet/roslyn/wiki/Roslyn%20Overview

About compiler plugins in Rust:

 In a nutshell Rust plugin's let you write normal imperative 
 run-time type-checked Rust code that is executed at 
 compile-time on the AST and they let you do absolutely anything 
 (File/Network I/O, launching threads, ...) including rewriting 
 the AST. People use it to extend the language "as a library": 
 implement coroutines, plain-Rust-to-GLSL libraries that allow 
 you to write shaders in Rust, GPGPU language extensions, and 
 also to write very powerful libraries: regex engines, 
 serialization libraries, database libraries that connect at 
 compile-time to the data-base to validate your SQL queries and 
 give you compile-time errors if they are invalid... EDSLs... 
 All in normal, imperative, run-time Rust code, without shadow 
 worlds (except for the AST API).


- some guy on reddit

.NET Compiler Platform ("Roslyn"): Analyzers and the Rise of 
Code-Aware Libraries:
https://www.youtube.com/watch?v=Ip6wrpYFHhE


 I guess it would make sense to extract parts of dmd into dub 
 packages. As a next step, dmd could use those packages instead 
 of duplicating code.


 Does it?  Which parts?  I'm afraid I don't see the benefit.

 -Wyatt


LOL x2
Jun 30 2016
next sibling parent default0 <Kevin.Labschek gmx.de> writes:
On Thursday, 30 June 2016 at 15:53:19 UTC, ZombineDev wrote:

 On Thursday, 30 June 2016 at 14:14:41 UTC, Wyatt wrote:

 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:

 Ok, seriously, it sounds like an awesome feat, but I don't 
 think it is necessary to put it into Phobos. First, a dub 
 package, please.


 Agree.  Does Java even have something like that?


 I'm afraid you couldn't be more wrong. Both Java and .NET have 
 many provide ways for generating and executing byte code at 
 run-time. Sometimes this the only way to implement something 
 efficiently when runtime reflection is needed. See for example:
 https://msdn.microsoft.com/en-us/library/mt654263.aspx,
 https://msdn.microsoft.com/en-us/library/mt654267.aspx,
 http://openjdk.java.net/groups/compiler/doc/package-overview/index.html,
 http://asm.ow2.org/,
 https://commons.apache.org/proper/commons-bcel/manual.html,
 https://github.com/cglib/cglib,
 http://jboss-javassist.github.io/javassist/,
 https://www.jetbrains.com/mps/index.html,
 https://docs.microsoft.com/en-us/dotnet/core/api/system.reflection.emit and
 https://www.postsharp.net/features

 Also almost every dynamic language has some sort of eval 
 function can be used to evaluate arbitrary code at run-time. 
 This has it's security and maintainability challenges, but 
 without any doubt there are situations where this is very 
 helpful.


 That's sort of the exemplar for "hopelessly overdone standard 
 library".


 LOL, I have never heard about a user complaining that a product 
 has too many features, as long as they don't get in the way. 
 Instead users complain when something is **not available**, 
 because that prevents them from getting their job done. What's 
 wrong with having a module in the standard library that you 
 personally won't use, but others will find helpful?

 There are many ways in which code-generation can interact with 
 the language runtime. By including it in the standard library, 
 we can ensure that it is thoroughly tested on all supported 
 platforms. Of course the other benefit is that it can be used 
 from other modules in the standard library for implementing 
 various optimizations (e.g. optimizing regex, linear algebra, 
 data base queries, etc.) Such functionality has been a huge 
 success for .NET. E.g. they enabled some advanced LINQ features 
 which are used under the hood of almost every .NET project.


 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?


 DMD shouldn't have to download things from the public internet 
 to do its job.


 I don't think you understood the question. The question is how 
 should DMD's code base be structured / modularized. Of course 
 after the DMD is compiled it shouldn't need to use the 
 internet, but that's not the point. The question is if it's a 
 good idea to split the project in small safe-contained reusable 
 packages. For example, that would allow linters to leverage the 
 compiler lexer and parser instead of implementing their own, 
 which often can't handle all language features.

 Another huge area is compiler plugins which are quite popular 
 in Rust and .NET
 https://doc.rust-lang.org/book/compiler-plugins.html
 https://github.com/dotnet/roslyn/wiki/Roslyn%20Overview

 About compiler plugins in Rust:

 In a nutshell Rust plugin's let you write normal imperative 
 run-time type-checked Rust code that is executed at 
 compile-time on the AST and they let you do absolutely 
 anything (File/Network I/O, launching threads, ...) including 
 rewriting the AST. People use it to extend the language "as a 
 library": implement coroutines, plain-Rust-to-GLSL libraries 
 that allow you to write shaders in Rust, GPGPU language 
 extensions, and also to write very powerful libraries: regex 
 engines, serialization libraries, database libraries that 
 connect at compile-time to the data-base to validate your SQL 
 queries and give you compile-time errors if they are 
 invalid... EDSLs... All in normal, imperative, run-time Rust 
 code, without shadow worlds (except for the AST API).


 - some guy on reddit

 .NET Compiler Platform ("Roslyn"): Analyzers and the Rise of 
 Code-Aware Libraries:
 https://www.youtube.com/watch?v=Ip6wrpYFHhE


 I guess it would make sense to extract parts of dmd into dub 
 packages. As a next step, dmd could use those packages 
 instead of duplicating code.


 Does it?  Which parts?  I'm afraid I don't see the benefit.

 -Wyatt


 LOL x2


My 2 cents on compiler plugins is that as cool and powerful and 
amazing as they sound they put quite the burden on their user. 
Compilers are complicated beasts. In order to write some code 
that correctly uses their API you first have to learn quite a lot 
of stuff (how to use their specific API, how their AST looks, 
what you can and can't do, why this particular error pops up 
etc). And from what I know in those languages, unlike in D, there 
is no **simple** go-to way of doing work at CT that you just need 
done.

I had the pleasure of writing a pseudo compiler extension for 
static interfaces using Roslyn, working out edge cases with 
generics and stuff was.... interesting. Lots of things to look up 
regarding symbol equality, how to compare what, what to cast what 
into to get at your specific information. I imagine Rust will be 
a similar story.
So while they are insanely powerful and nice, they aren't 
something you want to use often. They are sort of the thing you 
resort to as a last or second-to-last ditch effort at getting an 
idea implemented.

That said, having such a powerful, big library for code analysis 
at your disposal is amazing for tooling. My point is, it is in no 
way a replacement or even a compensation for the awesomeness that 
are D's **simple** mixins and template metaprogramming.
Jun 30 2016
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 30 June 2016 at 15:53:19 UTC, ZombineDev wrote:

 Also almost every dynamic language has some sort of eval 
 function can be used to evaluate arbitrary code at run-time.


And almost every language guideline suggest to never use it. When 
people have to use it, it usually ends in a library/language 
extension making it obsolete...


 LOL, I have never heard about a user complaining that a product 
 has too many features, as long as they don't get in the way.


Really? It is quite common to complain about standard libraries 
being stuck with obsolete features. Which is quite confusing to 
both beginners and intermediate programmers.


 data base queries, etc.) Such functionality has been a huge 
 success for .NET. E.g. they enabled some advanced LINQ features 
 which are used under the hood of almost every .NET project.



the IR is completely implementation defined.

A VM does not belong in a standard library. It is going to suck. 
If it doesn't suck today, then it will most certainly suck 
tomorrow or the day after.


 packages. For example, that would allow linters to leverage the 
 compiler lexer and parser instead of implementing their own, 
 which often can't handle all language features.


The standard library is for abstracting system specific aspects 
that libraries need to deal with. The most important aspect of a 
standard library is for writing libraries with a strong focus on 
consistency, not compilers or even applications. A standard 
library should have very high quality and performance 
requirements.

The D standard library is already too bloated to reach a high 
quality level and probably also suffer in the performance 
department.

Keep auxiliary stuff out, or D will never reach a competitive 
level.
Jun 30 2016
prev sibling next sibling parent jkpl <jkpl nowhere.de> writes:
On Thursday, 30 June 2016 at 10:28:03 UTC, Stefan Koch wrote:

 Hi,

 I recently had a breakthrough in my CTFE work.

 Though because habits die hard.
 I am writing the bytecode-engine in a CTFEable style.
 Therefore I can be used as a quite comfortable IR for CTFE 
 things as well.

 It should be fairly easy to generate a inline-asm-string from 
 the byteCode  at compiletime.
 And thereby creating the possibility of generating optimized 
 code at compile-time.

 I was wondering if such a thing would fit in phobos.

 Please share your opinions with me :)

 Regards,
 Stefan


No platform limitation is the rule of thumb.
Jun 30 2016
prev sibling parent ZombineDev <petar.p.kirov gmail.com> writes:
On Thursday, 30 June 2016 at 10:28:03 UTC, Stefan Koch wrote:

 Hi,

 I recently had a breakthrough in my CTFE work.

 Though because habits die hard.
 I am writing the bytecode-engine in a CTFEable style.
 Therefore I can be used as a quite comfortable IR for CTFE 
 things as well.

 It should be fairly easy to generate a inline-asm-string from 
 the byteCode  at compiletime.
 And thereby creating the possibility of generating optimized 
 code at compile-time.

 I was wondering if such a thing would fit in phobos.

 Please share your opinions with me :)

 Regards,
 Stefan


I think that's a great idea! It could be also useful for further 
optimizing ctRegex, among other things.

But I agree with others - please put it on DUB first so we can 
test it more widely before it's ready for Phobos. As we all know 
getting something accepted into Pbobos is not a walk in the park, 
unless you already have the perfect design: 
http://dlangcomicstrips.tumblr.com/image/128280515097
:D

BTW AFAIK  9il has done something similar for his BLAS library. 
His code generates inline LLVM IR at CT for LDC and generates 
vanilla D code for DMD.

Some realated discussions:
https://github.com/ldc-developers/ldc/issues/1438
https://github.com/ldc-developers/ldc/issues/1482
Jun 30 2016