www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DMD Frontend working in WebAssembly

reply hatf0 <harrison 0xcc.pw> writes:
Hi all,

I've just managed to get the full DMD front-end to work in 
WebAssembly (with skoppe's druntime fork). This doesn't do 
code-gen or anything (but it potentially could?), and has some 
OS-specific functionality stubbed out. No clue about GC -- 
haven't run into that issue, haven't thought about it yet!

You can find my work here if you're interested: 
https://github.com/hatf0/dmd-fe-wasm-wrapper

This repo also serves as a semi-decent guide on how to get 
started building files that target WebAssembly (specifically 
WASI), if you're into that.
Oct 14 2021
next sibling parent reply hatf0 <harrison 0xcc.pw> writes:
On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote:

 Hi all,

 I've just managed to get the full DMD front-end to work in 
 WebAssembly (with skoppe's druntime fork). This doesn't do 
 code-gen or anything (but it potentially could?), and has some 
 OS-specific functionality stubbed out. No clue about GC -- 
 haven't run into that issue, haven't thought about it yet!

 You can find my work here if you're interested: 
 https://github.com/hatf0/dmd-fe-wasm-wrapper

 This repo also serves as a semi-decent guide on how to get 
 started building files that target WebAssembly (specifically 
 WASI), if you're into that.


Oooh -- I totally forgot: for some reason, this doesn't work with 
the wasmer JS runtime. I haven't done any investigation into it, 
since it works just fine with the wasmer Rust runtime.

More work needs to be done before we can fully bring DMD to the 
browser, but this could totally open up the idea of having 
something like Swift Playgrounds within the browser (or perhaps 
some toys like AST inspection, or whatnot).
Oct 14 2021
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 14 October 2021 at 22:58:37 UTC, hatf0 wrote:

 On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote:

 Hi all,

 I've just managed to get the full DMD front-end to work in 
 WebAssembly (with skoppe's druntime fork).




Cool stuff. I recently picked up my wasm forks again and updated 
to wasi 12, ldc 1.25.0 and the accompanying druntime/phobos 
versions and got all tests to pass (those that I didn't version 
out). I am currently in the process of updating to latest ldc and 
latest druntime/phobos. Phobos has a bunch of merge conflicts, 
but it should be doable if I find a few hours.

Then there is the process of upstreaming a 5000+ line patch and I 
probably need some help there. I reckon the WASI bindings are 
pretty standalone so those should be easy to add, but there is a 
sprawl of additions across almost all files due to not supporting 
threads, fibers and catching exceptions.
Oct 15 2021
parent reply hatf0 <harrison 0xcc.pw> writes:
On Friday, 15 October 2021 at 07:28:55 UTC, Sebastiaan Koppe 
wrote:

 On Thursday, 14 October 2021 at 22:58:37 UTC, hatf0 wrote:

 On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote:

 Hi all,

 I've just managed to get the full DMD front-end to work in 
 WebAssembly (with skoppe's druntime fork).




 Cool stuff. I recently picked up my wasm forks again and 
 updated to wasi 12, ldc 1.25.0 and the accompanying 
 druntime/phobos versions and got all tests to pass (those that 
 I didn't version out). I am currently in the process of 
 updating to latest ldc and latest druntime/phobos. Phobos has a 
 bunch of merge conflicts, but it should be doable if I find a 
 few hours.
 Then there is the process of upstreaming a 5000+ line patch and 
 I probably need some help there. I reckon the WASI bindings are 
 pretty standalone so those should be easy to add, but there is 
 a sprawl of additions across almost all files due to not 
 supporting threads, fibers and catching exceptions.


Very cool! This all would not be possible with your wasm forks -- 
they are the saving grace here.

Regarding upstreaming the patch... that will be rather 
challenging, but I would not mind contributing. I definitely 
think the WASI bindings (if not all of the files in 
`core.sys.wasi.*`) should be upstreamed first, as they're the 
simplest to implement, and then the various druntime patches 
should follow.

GC also needs some investigation (or malloc), as I keep get 
spurious OOM errors. Could be because dmd is one hell of a memory 
hog, but who knows.
Oct 15 2021
next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Friday, 15 October 2021 at 11:52:24 UTC, hatf0 wrote:

 Very cool! This all would not be possible with your wasm
 forks -- they are the saving grace here.


Nice that you got it running.


 GC also needs some investigation (or malloc), as I keep get 
 spurious OOM errors. Could be because dmd is one hell of a 
 memory hog, but who knows.


So far I haven't hit those when running druntime/phobos tests, 
but they could be caused by pointers being hidden from GC because 
they sit on the wasm stack/registers.

The only way to solve that would be to (somehow) have LDC put 
them on the shadow stack whenever they aren't. I know of some 
experiments where they did this with a boehm style GC on 
emscripten. Maybe I need to look more into how other languages 
(like go) handle this.
Oct 15 2021
prev sibling parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Friday, 15 October 2021 at 11:52:24 UTC, hatf0 wrote:

 On Friday, 15 October 2021 at 07:28:55 UTC, Sebastiaan Koppe 
 wrote:

 [...]


 Very cool! This all would not be possible with your wasm forks 
 -- they are the saving grace here.

 Regarding upstreaming the patch... that will be rather 
 challenging, but I would not mind contributing. I definitely 
 think the WASI bindings (if not all of the files in 
 `core.sys.wasi.*`) should be upstreamed first, as they're the 
 simplest to implement, and then the various druntime patches 
 should follow.

 GC also needs some investigation (or malloc), as I keep get 
 spurious OOM errors. Could be because dmd is one hell of a 
 memory hog, but who knows.


Btw, why is dmd a memory hog?
Oct 15 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 15 October 2021 at 15:19:26 UTC, Imperatorn wrote:

 Btw, why is dmd a memory hog?


Because it doesn't free memory. It just allocates, and relies on 
the OS to clean up at process exit.

See https://www.digitalmars.com/articles/b87.html
Oct 15 2021
parent reply max haughton <maxhaton gmail.com> writes:
On Friday, 15 October 2021 at 21:58:29 UTC, Paul Backus wrote:

 On Friday, 15 October 2021 at 15:19:26 UTC, Imperatorn wrote:

 Btw, why is dmd a memory hog?


 Because it doesn't free memory. It just allocates, and relies 
 on the OS to clean up at process exit.

 See https://www.digitalmars.com/articles/b87.html


lowmem actually isn't much better sometimes. There are issues 
with references being held and confusing the GC, also.

You can see this, if you use either massif or enable the GC. The 
peak heap collapses but at not much expense in runtime. Sometimes 
however, the GC makes no difference - I noticed this when 
benchmarking AliasAssign, you'd expect the old assigned-to memory 
to be freed but this doesn't seem to be the case empirically.
Oct 15 2021
parent Elronnd <elronnd elronnd.net> writes:
On Saturday, 16 October 2021 at 01:08:11 UTC, max haughton wrote:

 issues with references being held and confusing the GC, also.


A reminder (if one were needed) that one can still have memory 
leaks with GC :)
Oct 16 2021
prev sibling next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote:

 Hi all,

 I've just managed to get the full DMD front-end to work in 
 WebAssembly (with skoppe's druntime fork). This doesn't do 
 code-gen or anything (but it potentially could?), and has some 
 OS-specific functionality stubbed out. No clue about GC -- 
 haven't run into that issue, haven't thought about it yet!

 You can find my work here if you're interested: 
 https://github.com/hatf0/dmd-fe-wasm-wrapper

 This repo also serves as a semi-decent guide on how to get 
 started building files that target WebAssembly (specifically 
 WASI), if you're into that.


Nice work guys ☀️
Oct 15 2021
prev sibling parent reply WebFreak001 <d.forum webfreak.org> writes:
On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote:

 Hi all,

 I've just managed to get the full DMD front-end to work in 
 WebAssembly (with skoppe's druntime fork). This doesn't do 
 code-gen or anything (but it potentially could?), and has some 
 OS-specific functionality stubbed out. No clue about GC -- 
 haven't run into that issue, haven't thought about it yet!

 You can find my work here if you're interested: 
 https://github.com/hatf0/dmd-fe-wasm-wrapper

 This repo also serves as a semi-decent guide on how to get 
 started building files that target WebAssembly (specifically 
 WASI), if you're into that.


neat! Given the low footprint of WASM apps maybe this could maybe 
be used to have small isolated, cross-platform DMD tools that run 
one-shot? Could for example run the WASM binaries from the IDE 
and because it's WASM running in the own process, with memory 
that we can free, it avoids low startup times, especially on 
windows.

Do we have some WASM runner in D?
Oct 15 2021
parent hatf0 <harrison 0xcc.pw> writes:
On Friday, 15 October 2021 at 09:54:06 UTC, WebFreak001 wrote:

 On Thursday, 14 October 2021 at 22:56:07 UTC, hatf0 wrote:

 Hi all,

 I've just managed to get the full DMD front-end to work in 
 WebAssembly (with skoppe's druntime fork). This doesn't do 
 code-gen or anything (but it potentially could?), and has some 
 OS-specific functionality stubbed out. No clue about GC -- 
 haven't run into that issue, haven't thought about it yet!

 You can find my work here if you're interested: 
 https://github.com/hatf0/dmd-fe-wasm-wrapper

 This repo also serves as a semi-decent guide on how to get 
 started building files that target WebAssembly (specifically 
 WASI), if you're into that.


 neat! Given the low footprint of WASM apps maybe this could 
 maybe be used to have small isolated, cross-platform DMD tools 
 that run one-shot? Could for example run the WASM binaries from 
 the IDE and because it's WASM running in the own process, with 
 memory that we can free, it avoids low startup times, 
 especially on windows.


Definitely -- the sky is the limit at this point. Once I figure 
out the funkiness that's going on with the JS WASI runtimes, I've 
thought of doing things like having ddoc generation occurring at 
the edge (i.e. CloudFlare Workers). This would allow for 
dynamically generation of the documentation pages for Dub 
packages (or even, generating them on the client-side), and would 
mean less infrastructure complexity overall. Or hell, doing 
linting or other code analysis at package submission at the edge. 
There's *so* much that's possible since the full front-end is 
operable.


 Do we have some WASM runner in D?


We have https://code.dlang.org/packages/wasmer, which wraps the 
wasmer C runtime. This package /may/ need some tidying up (as 
wasmer is on v1.0.0 and this is still on a RC).
Oct 15 2021