www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Does a Interpretation-Engine fit in phobos ?

reply Stefan Koch <uplink.coder googlemail.com> writes:
Hi,

I recently had a breakthrough in my CTFE work.

Though because habits die hard.
I am writing the bytecode-engine in a CTFEable style.
Therefore I can be used as a quite comfortable IR for CTFE things 
as well.

It should be fairly easy to generate a inline-asm-string from the 
byteCode  at compiletime.
And thereby creating the possibility of generating optimized code 
at compile-time.

I was wondering if such a thing would fit in phobos.

Please share your opinions with me :)

Regards,
Stefan
Jun 30 2016
next sibling parent reply qznc <qznc web.de> writes:
On Thursday, 30 June 2016 at 10:28:03 UTC, Stefan Koch wrote:
 Hi,

 I recently had a breakthrough in my CTFE work.

 Though because habits die hard.
 I am writing the bytecode-engine in a CTFEable style.
 Therefore I can be used as a quite comfortable IR for CTFE 
 things as well.

 It should be fairly easy to generate a inline-asm-string from 
 the byteCode  at compiletime.
 And thereby creating the possibility of generating optimized 
 code at compile-time.

 I was wondering if such a thing would fit in phobos.

 Please share your opinions with me :)

 Regards,
 Stefan
Any sufficiently complicated C or Fortran [or D] program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. –Greenspun's tenth rule Ok, seriously, it sounds like an awesome feat, but I don't think it is necessary to put it into Phobos. First, a dub package, please. Off-topic: Is it possible/feasible/desirable to let dmd use dub packages? I guess it would make sense to extract parts of dmd into dub packages. As a next step, dmd could use those packages instead of duplicating code.
Jun 30 2016
next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:
 Off-topic: Is it possible/feasible/desirable to let dmd use dub 
 packages?
please, no. not everybody out there is dub fan.
Jun 30 2016
next sibling parent Seb <seb wilzba.ch> writes:
On Thursday, 30 June 2016 at 11:09:10 UTC, ketmar wrote:
 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:
 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?
please, no. not everybody out there is dub fan.
Off-topic: ketmar: if it's on dub (d's official package manager), it's easily visible to others and searchable. Moreover most people prefer to have a dependency manager to download the sources and compile it. Can't you just grab the referenced source repository manually? Please open a new thread if that doesn't work for you ;-)
Jun 30 2016
prev sibling parent reply lobo <swamplobo gmail.com> writes:
On Thursday, 30 June 2016 at 11:09:10 UTC, ketmar wrote:
 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:
 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?
please, no. not everybody out there is dub fan.
You can always use dub to fetch packages into ~/.dub/packages and build them. The artifacts are then available for whatever build system you choose for your own project. I don't use DUB for my own projects because it drops garbage all through my source tree. The day it can do out of source builds is the day I'll revisit the DUB bazaar. bye, lobo
Jun 30 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 30 June 2016 at 23:27:10 UTC, lobo wrote:
 On Thursday, 30 June 2016 at 11:09:10 UTC, ketmar wrote:
 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:
 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?
please, no. not everybody out there is dub fan.
You can always use dub to fetch packages into ~/.dub/packages and build them. The artifacts are then available for whatever build system you choose for your own project. I don't use DUB for my own projects because it drops garbage all through my source tree. The day it can do out of source builds is the day I'll revisit the DUB bazaar. bye, lobo
The Bytecode interpreter is now CTFEable :)
Jul 02 2016
parent reply Joakim =?UTF-8?B?QnLDpG5uc3Ryw7Zt?= <notfornow dev.null.com> writes:
On Saturday, 2 July 2016 at 08:11:48 UTC, Stefan Koch wrote:
 The Bytecode interpreter is now CTFEable :)
Good job Stefan. The PR looks really interesting. I'll be looking into using your engine I future projects so I hope that it will be available in some way or another :) // Joakim
Jul 02 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 2 July 2016 at 09:22:48 UTC, Joakim Brännström wrote:
 On Saturday, 2 July 2016 at 08:11:48 UTC, Stefan Koch wrote:
 The Bytecode interpreter is now CTFEable :)
Good job Stefan. The PR looks really interesting. I'll be looking into using your engine I future projects so I hope that it will be available in some way or another :) // Joakim
Good it hear. I just pulled out a lot of the generation functions. And made sure the "Disassembler" is ctfeable as well.
Jul 02 2016
prev sibling parent reply Wyatt <wyatt.epp gmail.com> writes:
On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:
 Ok, seriously, it sounds like an awesome feat, but I don't 
 think it is necessary to put it into Phobos. First, a dub 
 package, please.
Agree. Does Java even have something like that? That's sort of the exemplar for "hopelessly overdone standard library".
 Off-topic: Is it possible/feasible/desirable to let dmd use dub 
 packages?
DMD shouldn't have to download things from the public internet to do its job.
 I guess it would make sense to extract parts of dmd into dub 
 packages. As a next step, dmd could use those packages instead 
 of duplicating code.
Does it? Which parts? I'm afraid I don't see the benefit. -Wyatt
Jun 30 2016
parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Thursday, 30 June 2016 at 14:14:41 UTC, Wyatt wrote:
 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:
 Ok, seriously, it sounds like an awesome feat, but I don't 
 think it is necessary to put it into Phobos. First, a dub 
 package, please.
Agree. Does Java even have something like that?
I'm afraid you couldn't be more wrong. Both Java and .NET have many provide ways for generating and executing byte code at run-time. Sometimes this the only way to implement something efficiently when runtime reflection is needed. See for example: https://msdn.microsoft.com/en-us/library/mt654263.aspx, https://msdn.microsoft.com/en-us/library/mt654267.aspx, http://openjdk.java.net/groups/compiler/doc/package-overview/index.html, http://asm.ow2.org/, https://commons.apache.org/proper/commons-bcel/manual.html, https://github.com/cglib/cglib, http://jboss-javassist.github.io/javassist/, https://www.jetbrains.com/mps/index.html, https://docs.microsoft.com/en-us/dotnet/core/api/system.reflection.emit and https://www.postsharp.net/features Also almost every dynamic language has some sort of eval function can be used to evaluate arbitrary code at run-time. This has it's security and maintainability challenges, but without any doubt there are situations where this is very helpful.
 That's sort of the exemplar for "hopelessly overdone standard 
 library".
LOL, I have never heard about a user complaining that a product has too many features, as long as they don't get in the way. Instead users complain when something is **not available**, because that prevents them from getting their job done. What's wrong with having a module in the standard library that you personally won't use, but others will find helpful? There are many ways in which code-generation can interact with the language runtime. By including it in the standard library, we can ensure that it is thoroughly tested on all supported platforms. Of course the other benefit is that it can be used from other modules in the standard library for implementing various optimizations (e.g. optimizing regex, linear algebra, data base queries, etc.) Such functionality has been a huge success for .NET. E.g. they enabled some advanced LINQ features which are used under the hood of almost every .NET project.
 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?
DMD shouldn't have to download things from the public internet to do its job.
I don't think you understood the question. The question is how should DMD's code base be structured / modularized. Of course after the DMD is compiled it shouldn't need to use the internet, but that's not the point. The question is if it's a good idea to split the project in small safe-contained reusable packages. For example, that would allow linters to leverage the compiler lexer and parser instead of implementing their own, which often can't handle all language features. Another huge area is compiler plugins which are quite popular in Rust and .NET https://doc.rust-lang.org/book/compiler-plugins.html https://github.com/dotnet/roslyn/wiki/Roslyn%20Overview About compiler plugins in Rust:
 In a nutshell Rust plugin's let you write normal imperative 
 run-time type-checked Rust code that is executed at 
 compile-time on the AST and they let you do absolutely anything 
 (File/Network I/O, launching threads, ...) including rewriting 
 the AST. People use it to extend the language "as a library": 
 implement coroutines, plain-Rust-to-GLSL libraries that allow 
 you to write shaders in Rust, GPGPU language extensions, and 
 also to write very powerful libraries: regex engines, 
 serialization libraries, database libraries that connect at 
 compile-time to the data-base to validate your SQL queries and 
 give you compile-time errors if they are invalid... EDSLs... 
 All in normal, imperative, run-time Rust code, without shadow 
 worlds (except for the AST API).
- some guy on reddit .NET Compiler Platform ("Roslyn"): Analyzers and the Rise of Code-Aware Libraries: https://www.youtube.com/watch?v=Ip6wrpYFHhE
 I guess it would make sense to extract parts of dmd into dub 
 packages. As a next step, dmd could use those packages instead 
 of duplicating code.
Does it? Which parts? I'm afraid I don't see the benefit. -Wyatt
LOL x2
Jun 30 2016
next sibling parent default0 <Kevin.Labschek gmx.de> writes:
On Thursday, 30 June 2016 at 15:53:19 UTC, ZombineDev wrote:
 On Thursday, 30 June 2016 at 14:14:41 UTC, Wyatt wrote:
 On Thursday, 30 June 2016 at 10:36:44 UTC, qznc wrote:
 Ok, seriously, it sounds like an awesome feat, but I don't 
 think it is necessary to put it into Phobos. First, a dub 
 package, please.
Agree. Does Java even have something like that?
I'm afraid you couldn't be more wrong. Both Java and .NET have many provide ways for generating and executing byte code at run-time. Sometimes this the only way to implement something efficiently when runtime reflection is needed. See for example: https://msdn.microsoft.com/en-us/library/mt654263.aspx, https://msdn.microsoft.com/en-us/library/mt654267.aspx, http://openjdk.java.net/groups/compiler/doc/package-overview/index.html, http://asm.ow2.org/, https://commons.apache.org/proper/commons-bcel/manual.html, https://github.com/cglib/cglib, http://jboss-javassist.github.io/javassist/, https://www.jetbrains.com/mps/index.html, https://docs.microsoft.com/en-us/dotnet/core/api/system.reflection.emit and https://www.postsharp.net/features Also almost every dynamic language has some sort of eval function can be used to evaluate arbitrary code at run-time. This has it's security and maintainability challenges, but without any doubt there are situations where this is very helpful.
 That's sort of the exemplar for "hopelessly overdone standard 
 library".
LOL, I have never heard about a user complaining that a product has too many features, as long as they don't get in the way. Instead users complain when something is **not available**, because that prevents them from getting their job done. What's wrong with having a module in the standard library that you personally won't use, but others will find helpful? There are many ways in which code-generation can interact with the language runtime. By including it in the standard library, we can ensure that it is thoroughly tested on all supported platforms. Of course the other benefit is that it can be used from other modules in the standard library for implementing various optimizations (e.g. optimizing regex, linear algebra, data base queries, etc.) Such functionality has been a huge success for .NET. E.g. they enabled some advanced LINQ features which are used under the hood of almost every .NET project.
 Off-topic: Is it possible/feasible/desirable to let dmd use 
 dub packages?
DMD shouldn't have to download things from the public internet to do its job.
I don't think you understood the question. The question is how should DMD's code base be structured / modularized. Of course after the DMD is compiled it shouldn't need to use the internet, but that's not the point. The question is if it's a good idea to split the project in small safe-contained reusable packages. For example, that would allow linters to leverage the compiler lexer and parser instead of implementing their own, which often can't handle all language features. Another huge area is compiler plugins which are quite popular in Rust and .NET https://doc.rust-lang.org/book/compiler-plugins.html https://github.com/dotnet/roslyn/wiki/Roslyn%20Overview About compiler plugins in Rust:
 In a nutshell Rust plugin's let you write normal imperative 
 run-time type-checked Rust code that is executed at 
 compile-time on the AST and they let you do absolutely 
 anything (File/Network I/O, launching threads, ...) including 
 rewriting the AST. People use it to extend the language "as a 
 library": implement coroutines, plain-Rust-to-GLSL libraries 
 that allow you to write shaders in Rust, GPGPU language 
 extensions, and also to write very powerful libraries: regex 
 engines, serialization libraries, database libraries that 
 connect at compile-time to the data-base to validate your SQL 
 queries and give you compile-time errors if they are 
 invalid... EDSLs... All in normal, imperative, run-time Rust 
 code, without shadow worlds (except for the AST API).
- some guy on reddit .NET Compiler Platform ("Roslyn"): Analyzers and the Rise of Code-Aware Libraries: https://www.youtube.com/watch?v=Ip6wrpYFHhE
 I guess it would make sense to extract parts of dmd into dub 
 packages. As a next step, dmd could use those packages 
 instead of duplicating code.
Does it? Which parts? I'm afraid I don't see the benefit. -Wyatt
LOL x2
My 2 cents on compiler plugins is that as cool and powerful and amazing as they sound they put quite the burden on their user. Compilers are complicated beasts. In order to write some code that correctly uses their API you first have to learn quite a lot of stuff (how to use their specific API, how their AST looks, what you can and can't do, why this particular error pops up etc). And from what I know in those languages, unlike in D, there is no **simple** go-to way of doing work at CT that you just need done. I had the pleasure of writing a pseudo compiler extension for static interfaces using Roslyn, working out edge cases with generics and stuff was.... interesting. Lots of things to look up regarding symbol equality, how to compare what, what to cast what into to get at your specific information. I imagine Rust will be a similar story. So while they are insanely powerful and nice, they aren't something you want to use often. They are sort of the thing you resort to as a last or second-to-last ditch effort at getting an idea implemented. That said, having such a powerful, big library for code analysis at your disposal is amazing for tooling. My point is, it is in no way a replacement or even a compensation for the awesomeness that are D's **simple** mixins and template metaprogramming.
Jun 30 2016
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Thursday, 30 June 2016 at 15:53:19 UTC, ZombineDev wrote:
 Also almost every dynamic language has some sort of eval 
 function can be used to evaluate arbitrary code at run-time.
And almost every language guideline suggest to never use it. When people have to use it, it usually ends in a library/language extension making it obsolete...
 LOL, I have never heard about a user complaining that a product 
 has too many features, as long as they don't get in the way.
Really? It is quite common to complain about standard libraries being stuck with obsolete features. Which is quite confusing to both beginners and intermediate programmers.
 data base queries, etc.) Such functionality has been a huge 
 success for .NET. E.g. they enabled some advanced LINQ features 
 which are used under the hood of almost every .NET project.
In Java and C#, the VM is the most stable language. In D/C++/C the IR is completely implementation defined. A VM does not belong in a standard library. It is going to suck. If it doesn't suck today, then it will most certainly suck tomorrow or the day after.
 packages. For example, that would allow linters to leverage the 
 compiler lexer and parser instead of implementing their own, 
 which often can't handle all language features.
The standard library is for abstracting system specific aspects that libraries need to deal with. The most important aspect of a standard library is for writing libraries with a strong focus on consistency, not compilers or even applications. A standard library should have very high quality and performance requirements. The D standard library is already too bloated to reach a high quality level and probably also suffer in the performance department. Keep auxiliary stuff out, or D will never reach a competitive level.
Jun 30 2016
prev sibling next sibling parent jkpl <jkpl nowhere.de> writes:
On Thursday, 30 June 2016 at 10:28:03 UTC, Stefan Koch wrote:
 Hi,

 I recently had a breakthrough in my CTFE work.

 Though because habits die hard.
 I am writing the bytecode-engine in a CTFEable style.
 Therefore I can be used as a quite comfortable IR for CTFE 
 things as well.

 It should be fairly easy to generate a inline-asm-string from 
 the byteCode  at compiletime.
 And thereby creating the possibility of generating optimized 
 code at compile-time.

 I was wondering if such a thing would fit in phobos.

 Please share your opinions with me :)

 Regards,
 Stefan
No platform limitation is the rule of thumb.
Jun 30 2016
prev sibling parent ZombineDev <petar.p.kirov gmail.com> writes:
On Thursday, 30 June 2016 at 10:28:03 UTC, Stefan Koch wrote:
 Hi,

 I recently had a breakthrough in my CTFE work.

 Though because habits die hard.
 I am writing the bytecode-engine in a CTFEable style.
 Therefore I can be used as a quite comfortable IR for CTFE 
 things as well.

 It should be fairly easy to generate a inline-asm-string from 
 the byteCode  at compiletime.
 And thereby creating the possibility of generating optimized 
 code at compile-time.

 I was wondering if such a thing would fit in phobos.

 Please share your opinions with me :)

 Regards,
 Stefan
I think that's a great idea! It could be also useful for further optimizing ctRegex, among other things. But I agree with others - please put it on DUB first so we can test it more widely before it's ready for Phobos. As we all know getting something accepted into Pbobos is not a walk in the park, unless you already have the perfect design: http://dlangcomicstrips.tumblr.com/image/128280515097 :D BTW AFAIK 9il has done something similar for his BLAS library. His code generates inline LLVM IR at CT for LDC and generates vanilla D code for DMD. Some realated discussions: https://github.com/ldc-developers/ldc/issues/1438 https://github.com/ldc-developers/ldc/issues/1482
Jun 30 2016