www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Proposed strategy to support WASM in D

reply AnimusPEXUS <animuspexus protonmail.com> writes:
0. probably two different approaches to supporting wasm is needed 
simultatiously:
0.1: [wasm module] + [js glue] -- this is current approach, 
similar to golang's
0.2: wasi support - this approach means what wasi platforms 
provide interface for wasi application to interact with the system

I'm talking here about [wasm module] + [js glue] approach

1. this approach mean the task have to be separated to two 
subtasks:
1.1 js glue code, which allows wasm code to interact with js 
environment.
1.2 druntime have to be written, so Dlang code would be 
compilable with ldc (or yet better with dmd it self) with wasm 
triplet, without --betterC option, like any usual D program.


not so long ago I've started new project 
https://github.com/AnimusPEXUS/dwasmjs and I've wrote some basic 
js code to work with js values from inside D ( 
https://github.com/AnimusPEXUS/dwasmjs/blob/master/source/dwasmjs/dwasmjsi.js )
it's far from being complete. but the general Idea - is to store js values on
js side of js<>d glue code. (also I think reflect module of js could be used to
improve js interactions, to avoid eval() function use, as eval would slow
everything down)

also, to address memory transfers from js to d (wasm) - I've 
figured out what js side can ask D side to allocate memory arrays 
as needed and js should write bytes back to pointer passed by D 
side.

this js side of glue is working with identifiers (randomly 
generated unique values). if D side needs to get some string - D 
side calls function on JS side with id of value from which D side 
wants to get string, and D side also passes pointer to 
preallocated array (D side gets size needed for array by calling 
another function).

in my code I've stacked with need to allocate memory inside of 
wasm module instance. I've seen ADR's mini-druntime for wasm and 
seen how he used ldc's special fields to calculate addressing for 
memory allocation and used special ldc function to grow wasm blob 
(I think, those function can be called from standard WebAssembly 
module of JS, avoiding need in special ldc functions)

vvv
so basically, the main problem now in literally is to write 
WebAssembly version of druntime. gluing it to js is not so 
problematic.
^^^

--------------

as for WASI support, I've didn't read documentation yet, but I 
imagine it provides some binary interface so it can be used to 
interact with the system, so WASI, probably should be separate 
version of druntime - separate from WebAssembly(64)

so currently I'm reading druntime and trying to figureout how to 
modify it to be compilable in wasm

--------------

additionally:

1. LDC have to support unsigned integers in wasm function 
parameters, as currently wasm generated by ldc puts i32 type on 
all parameters.

2. we have to answer question: does stdc parts of druntime also 
should be ported to WebAssembly, as I've discovered what, looks 
like druntime requires struct tm from sys.posix.time module
Nov 17 2022
next sibling parent AnimusPEXUS <animuspexus protonmail.com> writes:
also I've collected some useful links for various WASM and JS 
resources related to WASM<>JS integration for ppl less to Google. 
here

* 
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions
* 
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API
* 
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Browser_support_for_JavaScript_APIs
* https://developer.mozilla.org/en-US/docs/Web
* https://developer.mozilla.org/en-US/docs/Web/API
* https://developer.mozilla.org/en-US/docs/WebAssembly
* 
https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface
* 
https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Memory/grow
* 
https://developer.mozilla.org/en-US/docs/WebAssembly/Using_the_JavaScript_API
* https://developer.mozilla.org/en-US/docs/Web/JavaScript
* 
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference
* https://github.com/adamdruppe/webassembly
* 
https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-documents.md
* 
https://github.com/bytecodealliance/wasmtime/blob/main/docs/WASI-intro.md
* https://github.com/ldc-developers/ldc
* https://github.com/WebAssembly/proposals
* 
https://github.com/WebAssembly/tool-conventions/blob/main/BasicCABI.md
* 
https://github.com/WebAssembly/tool-conventions/blob/main/Linking.md
* 
https://github.com/WebAssembly/WASI/blob/main/phases/snapshot/docs.md
* https://lld.llvm.org/WebAssembly.html
* https://llvm.org/doxygen/namespacellvm_1_1wasm.html
* https://llvm.org/doxygen/namespaces.html
* 
https://rob-blackbourn.github.io/blog/webassembly/wasm/strings/javascript/c/libc/wasm-libc/clang/2020/06/20/wasm-string-passing.html
* 
https://stackoverflow.com/questions/41353389/how-can-i-return-a-javascript-string-from-a-webassembly-function
* https://wasi.dev/
* https://webassembly.github.io/spec/core/
* https://webassembly.github.io/spec/core/exec/index.html
* 
https://webassembly.github.io/spec/core/exec/instructions.html#memory-instructions
* https://webassembly.github.io/spec/js-api/index.html
* https://webassembly.github.io/spec/js-api/#memories
* https://webassembly.github.io/spec/web-api/index.html
* https://webassembly.org/specs/
* 
https://wiki.dlang.org/Generating_WebAssembly_with_LDC#Calling_external_functions
* https://wiki.dlang.org/LDC-specific_language_changes
* https://wiki.dlang.org/Runtime_internals
* https://wiki.dlang.org/Using_LDC
* https://www.llvm.org/docs/ExtendingLLVM.html
* https://www.w3.org/TR/wasm-core/
Nov 17 2022
prev sibling next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 18 November 2022 at 00:55:57 UTC, AnimusPEXUS wrote:
 vvv
 so basically, the main problem now in literally is to write 
 WebAssembly version of druntime. gluing it to js is not so 
 problematic.
 ^^^
I took Sebastiaan's thing from two years ago and applied it to new druntime here: https://github.com/adamdruppe/dmd/tree/wasm I haven't tried actually compiling it yet though, I've been pretty swamped with work lately. When the new ldc comes out, I'll get Sebastiaan's documentation together and give it a try.
 interact with the system, so WASI, probably should be separate 
 version of druntime - separate from WebAssembly(64)
WebAssembly is a kind of CPU and WASI is a kind of C library. So it should be two separate versions kinda like how there's version(X86) and version(Windows) and version(CRuntime_Microsoft) all independent (though often used together you don't have to).
Nov 17 2022
parent reply AnimusPEXUS <animuspexus protonmail.com> writes:
On Friday, 18 November 2022 at 02:00:40 UTC, Adam D Ruppe wrote:
 On Friday, 18 November 2022 at 00:55:57 UTC, AnimusPEXUS wrote:

 I took Sebastiaan's thing from two years ago and applied it to 
 new druntime here:

 https://github.com/adamdruppe/dmd/tree/wasm
Have question(s) on Sebastiaan's wasm. 1. Sebastiaan, does your wasm realization concentrates primarily on wasi interfacing, or will it allows wasm usage in [wasm]+[js glue] fashion? 2. if [wasm]+[js glue] fashion is not supported by Sebastiaan's wasm realization, then I, I think need to concentrate on [wasm]+[js glue] variant. so if this is the case, then how to separate [wasm wasi] version from [wasm]+[js glue] version in druntime. 3. if [wasm]+[js glue] doesn't have place in druntime, then how to properly organize forking/patching of druntime?
Nov 24 2022
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 24 November 2022 at 20:22:43 UTC, AnimusPEXUS wrote:
 Have question(s) on Sebastiaan's wasm.

 1. Sebastiaan, does your wasm realization concentrates 
 primarily on wasi interfacing, or will it allows wasm usage in 
 [wasm]+[js glue] fashion?
DRuntime depends on libc. So if you want to use DRuntime in WASM you need a WASM implementation of libc. That is what WASI-libc provides. Why WASI? Well, it is the interface many WASM runtimes use. That is all my DRuntime 'port' does.
 2. if [wasm]+[js glue] fashion is not supported by Sebastiaan's 
 wasm realization, then I, I think need to concentrate on 
 [wasm]+[js glue] variant. so if this is the case, then how to 
 separate [wasm wasi] version from [wasm]+[js glue] version in 
 druntime.
Interfacing with JS is out of the scope of the DRuntime WASM/WASI port. In the Spasm library I have a bindgen tool that spits out D/JS glue code based on web IDL files. It allows you to call pretty much every web api out there. Before my current job I briefly worked on one that creates D/JS glue code for typescript libraries. It isn't finished, but it shows its viable.
Nov 25 2022
prev sibling next sibling parent reply Johan <j j.nl> writes:
On Friday, 18 November 2022 at 00:55:57 UTC, AnimusPEXUS wrote:
 
 1. LDC have to support unsigned integers in wasm function 
 parameters, as currently wasm generated by ldc puts i32 type on 
 all parameters.
Can you give an example of what goes wrong? `i32` is LDC/LLVM's type of signed _and_ unsigned integers, there is no distinction on LLVM IR level. -Johan
Nov 17 2022
parent reply AnimusPEXUS <animuspexus protonmail.com> writes:
On Friday, 18 November 2022 at 07:06:08 UTC, Johan wrote:
 On Friday, 18 November 2022 at 00:55:57 UTC, AnimusPEXUS wrote:
 
Can you give an example of what goes wrong? `i32` is LDC/LLVM's type of signed _and_ unsigned integers, there is no distinction on LLVM IR level. -Johan
for example, if I export function uint fname(uint p1, uint p2), the resulting wasm code treats values as signed. but actually I don't know if it possible for wasm to really know I wast to pass unsigned value.
Nov 17 2022
parent Paulo Pinto <pjmlp progtools.org> writes:
On Friday, 18 November 2022 at 07:28:33 UTC, AnimusPEXUS wrote:
 On Friday, 18 November 2022 at 07:06:08 UTC, Johan wrote:
 On Friday, 18 November 2022 at 00:55:57 UTC, AnimusPEXUS wrote:
 
Can you give an example of what goes wrong? `i32` is LDC/LLVM's type of signed _and_ unsigned integers, there is no distinction on LLVM IR level. -Johan
for example, if I export function uint fname(uint p1, uint p2), the resulting wasm code treats values as signed. but actually I don't know if it possible for wasm to really know I wast to pass unsigned value.
It looks like that if you look at the number, but in reality WASM makes no distiction, it is up for the application themselves to use the appropriate set of opcodes.
 The class  defines uninterpreted integers, whose signedness 
 interpretation can vary depending on context. In the abstract 
 syntax, they are represented as unsigned values. However, some 
 operations convert them to signed based on a two’s complement 
 interpretation.
https://webassembly.github.io/spec/core/syntax/values.html#integers https://webassembly.github.io/spec/core/syntax/instructions.html#syntax-sx
Nov 18 2022
prev sibling parent reply Pierce Ng <pierce samadhiweb.com> writes:
On Friday, 18 November 2022 at 00:55:57 UTC, AnimusPEXUS wrote:
 0.2: wasi support - this approach means what wasi platforms 
 provide interface for wasi application to interact with the 
 system
WASI is like a libc interface for Wasm and [WASI libc](https://github.com/WebAssembly/wasi-libc) is an implementation. A Wasm program calls WASI to access libc/OS-like functionality provided by the Wasm runtime, which could be the web browser or other runtimes. Earlier this year, for the fantasy game console [TIC-80](https://github.com/nesbox/TIC-80), I implemented the [D-Wasm template](https://github.com/nesbox/TIC-80/tree/main/templates/d). However, I'm stuck actually implementing games in D, (IHIRC) because of a symbol clash between TIC-80's time() API function and WASI libc's time(). For comparison and contrast, with the Zig TIC-80 Wasm interface, Zig source code calls tic.time() for the TIC-80 time() function and thus no symbol conflict. Here's the code snippet: ``` // D void TIC() { cls(0); for (int x=0; x<240; x+=28) { for (int y=0; y<136; y+=28) { cx = 12*sinf(time()/30000*(x+y+1)); // <========== time() symbol conflict cy = 12*cosf(time()/30000*(x+y+1)); // <========== ditto Pir(x, y, 25, 25, x+cx, y+cy); } } } // Zig export fn TIC() void { // some var initialization removed tic.cls(3); while (ax <= 240) { fx = intToFloat(f32, ax); ay = 0; while (ay <= 136) { fy = intToFloat(f32, ay); ca = 12*std.math.sin(tic.time()/30000*(fx+fy+1)); // tic.time() cb = 12*std.math.cos(tic.time()/30000*(fx+fy+1)); // no conflict Pir(fx, fy, 25, 25, fx+ca, fy+cb); ay += 28; } ax += 28; } } ``` Without modifying TIC-80 source code, is there a way to do some 'namespace' thingie for D like how Zig does it? More generally, as a D newbie, the confusion for me was (still is): Should I be writing D programs (whether for TIC-80 or more mundane platforms like Linux) in D, or in BetterC?
Nov 18 2022
next sibling parent AnimusPEXUS <animuspexus protonmail.com> writes:
On Saturday, 19 November 2022 at 01:16:00 UTC, Pierce Ng wrote:

 Without modifying TIC-80 source code, is there a way to do some 
 'namespace' thingie for D like how Zig does it?
probably you can try playing with [pragma(mangle)](https://dlang.org/spec/pragma.html#mangle) for this.
Nov 18 2022
prev sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Saturday, 19 November 2022 at 01:16:00 UTC, Pierce Ng wrote:
 Without modifying TIC-80 source code, is there a way to do some 
 'namespace' thingie for D like how Zig does it?
All D symbols are namespaced by their module name. You can expose the JS functions in different modules. I don't know your specific problem but it ought to just be a case of adjusting the imports on JS and using them through different modules in D.
 More generally, as a D newbie, the confusion for me was (still 
 is): Should I be writing D programs (whether for TIC-80 or more 
 mundane platforms like Linux) in D, or in BetterC?
Always use real D unless it is impractical to do so. No point going through suffering when you don't have to.
Nov 18 2022
parent Pierce Ng <pierce samadhiweb.com> writes:
On Saturday, 19 November 2022 at 01:57:21 UTC, Adam D Ruppe wrote:
 On Saturday, 19 November 2022 at 01:16:00 UTC, Pierce Ng wrote:
 Without modifying TIC-80 source code, is there a way to do 
 some 'namespace' thingie for D like how Zig does it?
All D symbols are namespaced by their module name. You can expose the JS functions in different modules.
Thanks. I misremembered, the trouble isn't with time() but with trying to use WASI libc's sinf() and cosf(). If I try to use D's std.math instead, the code doesn't compile in BetterC mode: ``` % make dub build --quiet --build release --compiler ldc2 --arch wasm32-unknown-unknown-wasm /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4433,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4435,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4437,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4537,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4539,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4541,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4558,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4560,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/math.d(4562,13): Error: undefined identifier `c_long` /home/pierce/pkg/ldc/include/d/core/stdc/fenv.d(875,5): Error: undefined identifier `fenv_t` /home/pierce/pkg/ldc/include/d/core/stdc/fenv.d(878,5): Error: undefined identifier `fexcept_t` /home/pierce/pkg/ldc/include/d/core/stdc/fenv.d(880,5): Error: undefined identifier `fexcept_t` /home/pierce/pkg/ldc/include/d/core/stdc/fenv.d(888,5): Error: undefined identifier `fenv_t` /home/pierce/pkg/ldc/include/d/core/stdc/fenv.d(890,5): Error: undefined identifier `fenv_t` /home/pierce/pkg/ldc/include/d/core/stdc/fenv.d(939,9): Error: undefined identifier `fenv_t` ldc2 failed with exit code 1. make: *** [Makefile:7: build] Error 2 ``` In TIC-80's case there is no JS. TIC-80 embeds the [wasm3](https://github.com/wasm3/wasm3) standalone runtime and makes its (TIC-80's) APIs available to Wasm programs (Well, TIC-80 also embeds Duktape, thus making it programmable with JS, but that's not relevant here.)
 Always use real D unless it is impractical to do so. No point 
 going through suffering when you don't have to.
Good point. Let me update D, TIC-80, etc. and try again.
Nov 18 2022